VBulletin 3.8 to Discourse on Docker Migration Test Take Two

OK.

Like we all do, we learn a lot from tests, test migrations, and so forth.

Today, I started from scratch on test migration 2, armed with a lot more knowledge,

The main differences are as follows:

• Installed discourse plugin ruby-bbcode-to-md before starting the install
• Modified vbulletin.rb to account for many (not all) of our custom bbcode, like our icode and. mods tags
• Modified vbulletin.rb to compensate for the fact that vb3 does not have a filedata table (like vb4 )
• Modified vbulletin.rb to compensate for an error in converting child forums to categories in discourse.
• Moved all avatars to the vb3 database from the file system before dumping the DB.

Running again, from scratch, vbulletin_neo.rb :

root@localhost-app:/var/www/discourse# su discourse -c 'bundle exec ruby script/import_scripts/vbulletin_neo.rb'
root:@localhost wants vb3
Loading existing groups...
Loading existing users...
Loading existing categories...
Loading existing posts...
Loading existing topics...

importing groups..
       20 / 20 (100.0%)  [1542 items/min]  ]  
importing users
      860 / 138144 (  0.6%)  [123 items/min]  in]  

Let's see how this goes....

PS: As a side note, when migrating like this, do not use the same username for your Discourse admin account as your legacy forum username (the forum you are migrating from) because if you do, there will be an issue importing your posts under your legacy forum username into discourse. In this "round two", I did not make that mistake

Anyway.... thing are chugging along again in "round two", from all the lessons learned from "round one".......

root@localhost-app:/var/www/discourse# su discourse -c 'bundle exec ruby script/import_scripts/vbulletin_neo.rb'
root:@localhost wants vb3
Loading existing groups...
Loading existing users...
Loading existing categories...
Loading existing posts...
Loading existing topics...

importing groups..
       20 / 20 (100.0%)  [1542 items/min]  ]  
importing users
     3745 / 138144 (  2.7%)  [117 items/min]  in]  

I will be surprised (pleasantly surprised) if this goes without flaws on "round two"....

Let's see how this go-around goes...

Estimated time to import / migration only the users again: 18 to 19 hours.

importing users
     8320 / 138144 (  6.0%)  [117 items/min]  in]  
     8431 / 138144 (  6.1%)  [117 items/min]  

I'm so glad I purged "inactive users with zero posts and zero likes" a year or so ago now

Note:

importing users
    42897 / 138144 ( 31.1%) 

Update:

It quit due to a broken ssh connect over the net, so I installed tmux and restarted in a tmux session.

Still chugging along:

importing users
    54764 / 138144 ( 39.6%)  

However, when I check in the admin control panel, the ruby-bbcode-to-md plugin does not appear.

I check the discourse ./plugins directory and ruby-bbcode-to-md is there and there are no errors when I rebuilt the discourse with ./launcher rebuild app

So, I'm a bit worried the ruby-bbcode-to-md plugin is not really installed properly

Not a good feeling about this plugin since it shows it installed in the plugin directory and there were on install errors in the log:

1024×473 59.1 KB

Also, oddly, I got this error on users migration a number of times:

  50287 / 138144 ( 36.4%)  [997 items/min]  Failed to create user id: 899992147, username: username1, email: blablah@mailinator.com                                                 
user errors: ["Primary email is not allowed from that email provider. Please use another email address."]  

So, I stopped the process and ran this mysql command against the vb3 dump:

update user set email = replace(email,'mailinator','mailgator') where email like '%mailinator%';

and restarted the migration and those errors went away... LOL

I cannot see any reason for this "email blocking" in the admin settings... which worries me a bit, hidden "rules" and "blocks" and "suspects" which are not identifiable in the admin settings :yikes:

Nine hours after starting this, the user migration is a bit over halfway done:

importing users
  80286 / 138144 ( 58.1%)

But this will not be the last test run, because I am fairly certain the bbcode to markdown plugin is not working as it should.

Let's see when it is all "baked" again..... what happened.

Those mail issues should have arised when registering a user, not during a migration..
Smells like bad cooking.

Oh, so just 18 hours of user migration alone?
Geez, that sounds like alot of data, meanwhile, it shows 'just 138k' , so that's like...
9 user per hour...
Sounds utterly slow...

If it is just the user data (not including posts) that is.

importing users
 
   119457 / 138144 ( 86.5%)  [347 items/min] 

Still chugg'n along:

import topics
164667 / 240510 ( 68.5%)

My modification from vbulletin.rb to vbulletin_neo.rb has worked so far and fixed the importing of forums to categories was done correctly this round.... now all forums have been converted to categories .

Still need some tweaking, but making progress...

Discourse seems to "reject" some user ids based on email address, but it's not clear how to adjust this.

import topics (final this run)
   236002 / 240510 ( 98.1%) 

Now working on:

importing posts...
    31136 / 651550 (  4.8%) 

Ran this against main forum database because of problem porting:

mysql> update user set email = replace(email,'mailinator','123456789ABCDE') where email like '%mailinator%';
Query OK, 115 rows affected (0.24 sec)
Rows matched: 115  Changed: 115  Warnings: 0

This does not seem to effect one active member only. For that member, please contact me with your new email address::

mysql> select username, lastactivity, FROM_UNIXTIME(lastactivity) as time, posts  from user where email like '%123456789ABCDE%' order by lastactivity desc;
+-----------------+--------------+---------------------+-------+
| username        | lastactivity | time                | posts |
+-----------------+--------------+---------------------+-------+
| foad            |   1578625700 | 2020-01-09 21:08:20 |    10 |
| Tibetano        |   1557180162 | 2019-05-06 17:02:42 |     0 |
| jenny-hud       |   1556347768 | 2019-04-27 01:49:28 |     0 |
| foobar1234      |   1550774615 | 2019-02-21 12:43:35 |     0 |
| dukiteh125      |   1511264391 | 2017-11-21 05:39:51 |     0 |
| Gregorylek      |   1487379968 | 2017-02-17 19:06:08 |     0 |
| supaflygy       |   1468750696 | 2016-07-17 05:18:16 |     1 |
| AngelikaSh      |   1448173168 | 2015-11-22 00:19:28 |     0 |
| threesixtyfive  |   1441735701 | 2015-09-08 13:08:21 |     2 |
| palmfrond       |   1437107050 | 2015-07-16 23:24:10 |     2 |
| mail1234        |   1433129936 | 2015-05-31 22:38:56 |     1 |
| coolkid         |   1417703668 | 2014-12-04 08:34:28 |    84 |
| jablonski12356  |   1416450564 | 2014-11-19 20:29:24 |     0 |
| ekbaazigar      |   1409003803 | 2014-08-25 16:56:43 |     4 |
| xuro            |   1406927712 | 2014-08-01 16:15:12 |     3 |
| huli_ka         |   1406289056 | 2014-07-25 06:50:56 |     0 |
| DoohanMcGirk    |   1402612539 | 2014-06-12 17:35:39 |     4 |
| ijustneeda      |   1398598080 | 2014-04-27 06:28:00 |    14 |
| CharlieRi       |   1393798942 | 2014-03-02 16:22:22 |     0 |
| catweasel@      |   1387983215 | 2013-12-25 08:53:35 |     0 |
| apackofwankers  |   1386514440 | 2013-12-08 08:54:00 |     0 |
| HarryFish       |   1384424680 | 2013-11-14 04:24:40 |     0 |
| unknownn        |   1374375480 | 2013-07-20 21:58:00 |     3 |
| cseng214        |   1373634453 | 2013-07-12 08:07:33 |     0 |
| snobbycharlton  |   1371472564 | 2013-06-17 07:36:04 |     0 |
| JennaH83        |   1370994551 | 2013-06-11 18:49:11 |     0 |
| DorieUmq        |   1370043948 | 2013-05-31 18:45:48 |     0 |
| LidiaRous       |   1368860327 | 2013-05-18 01:58:47 |     0 |
| CarriUQO        |   1368625281 | 2013-05-15 08:41:21 |     0 |
| HaleyMerr       |   1367576269 | 2013-05-03 05:17:49 |     0 |
| GwenHuggi       |   1367504147 | 2013-05-02 09:15:47 |     0 |
| TomFord1        |   1365528707 | 2013-04-09 12:31:47 |     3 |
| pericopericone  |   1361488097 | 2013-02-21 17:08:17 |     1 |
| oartns          |   1360192834 | 2013-02-06 17:20:34 |     0 |
| MasonJ          |   1360034451 | 2013-02-04 21:20:51 |     2 |
| jagoodwi        |   1359573506 | 2013-01-30 13:18:26 |     0 |
| joenewbie       |   1351617861 | 2012-10-30 12:24:21 |     0 |
| AcidTripz       |   1348337167 | 2012-09-22 13:06:07 |     0 |
| majortomtom     |   1342089610 | 2012-07-12 05:40:10 |     0 |
| koteshwor       |   1341644871 | 2012-07-07 02:07:51 |     0 |
| Tyler Aaron     |   1339530733 | 2012-06-12 14:52:13 |     2 |
| meightr         |   1336902792 | 2012-05-13 04:53:12 |     2 |
| richard78       |   1333452930 | 2012-04-03 06:35:30 |     1 |
| elohssa         |   1330044204 | 2012-02-23 18:43:24 |     3 |
| roby2411        |   1329986033 | 2012-02-23 02:33:53 |    12 |
| bobanite        |   1328802487 | 2012-02-09 09:48:07 |     0 |
| nugmebot        |   1326735854 | 2012-01-16 11:44:14 |     0 |
| chipinmybrain   |   1326602936 | 2012-01-14 22:48:56 |     1 |
| kgtsch          |   1326220373 | 2012-01-10 12:32:53 |     0 |
| justgoogleit    |   1325915800 | 2012-01-06 23:56:40 |     1 |
| chrisperry      |   1325549419 | 2012-01-02 18:10:19 |     8 |
| cdwdc           |   1322450384 | 2011-11-27 21:19:44 |     0 |
| gerdl           |   1317982481 | 2011-10-07 05:14:41 |     0 |
| jstilby         |   1315981589 | 2011-09-14 01:26:29 |     2 |
| hunterm         |   1311067040 | 2011-07-19 04:17:20 |     0 |
| TomTest         |   1309281462 | 2011-06-28 12:17:42 |     0 |
| nena_redbalon   |   1308934248 | 2011-06-24 11:50:48 |     2 |
| ThePistonDoctor |   1308672390 | 2011-06-21 11:06:30 |     6 |
| meuser          |   1308069000 | 2011-06-14 11:30:00 |     5 |
| blahblah123     |   1300283349 | 2011-03-16 08:49:09 |     0 |
| asdasdasdasdada |   1298756171 | 2011-02-26 15:36:11 |     0 |
| _light_         |   1296569679 | 2011-02-01 08:14:39 |     3 |
| maria_florencia |   1295530010 | 2011-01-20 07:26:50 |     1 |
| </kida>   |   1295299434 | 2011-01-17 15:23:54 |     4 |
| linber2880      |   1293468261 | 2010-12-27 10:44:21 |     3 |
| TroubleNow345   |   1293388140 | 2010-12-26 12:29:00 |     2 |
| datinskys       |   1293096529 | 2010-12-23 03:28:49 |     0 |
| sylar           |   1292717118 | 2010-12-18 18:05:18 |     1 |
| xen0n           |   1286993566 | 2010-10-13 13:12:46 |     0 |
| cprogdude       |   1286844464 | 2010-10-11 19:47:44 |     3 |
| pattu           |   1285287958 | 2010-09-23 19:25:58 |     3 |
| scriptfriend    |   1285283121 | 2010-09-23 18:05:21 |     3 |
| andra           |   1281085682 | 2010-08-06 04:08:02 |     0 |
| grandguest      |   1279554122 | 2010-07-19 10:42:02 |     4 |
| nicosarasa      |   1274988382 | 2010-05-27 14:26:22 |     0 |
| joodas          |   1274240403 | 2010-05-18 22:40:03 |     1 |
| castexyz        |   1271154599 | 2010-04-13 05:29:59 |     1 |
| rohitmitra5752  |   1270641508 | 2010-04-07 06:58:28 |     0 |
| ayu             |   1269477686 | 2010-03-24 19:41:26 |     1 |
| fbauto1         |   1268007442 | 2010-03-07 18:17:22 |     1 |
| silk600         |   1265973556 | 2010-02-12 05:19:16 |     1 |
| tester123       |   1264705175 | 2010-01-28 12:59:35 |     0 |
| maneth          |   1259987223 | 2009-12-04 22:27:03 |     1 |
| linuca          |   1259941435 | 2009-12-04 09:43:55 |     3 |
| lnxer           |   1259448014 | 2009-11-28 16:40:14 |     1 |
| zruasotcjpnjxw  |   1258688935 | 2009-11-19 21:48:55 |     0 |
| pru             |   1256045844 | 2009-10-20 08:37:24 |     4 |
| procreator      |   1255618625 | 2009-10-15 09:57:05 |     1 |
| hobbers         |   1245341071 | 2009-06-18 11:04:31 |     2 |
| romantico       |   1236377323 | 2009-03-06 16:08:43 |     0 |
| mathis          |   1234543606 | 2009-02-13 10:46:46 |     2 |
| ackbarr         |   1234026113 | 2009-02-07 11:01:53 |     3 |
| xadamz23        |   1231967591 | 2009-01-14 15:13:11 |    31 |
| d34dh0r53       |   1230662517 | 2008-12-30 12:41:57 |     2 |
| bobknob         |   1228116970 | 2008-12-01 01:36:10 |     2 |
| verno           |   1227952536 | 2008-11-29 03:55:36 |    14 |
| renmaverick     |   1226212425 | 2008-11-09 00:33:45 |     3 |
| user19190989    |   1224347984 | 2008-10-18 11:39:44 |     1 |
| bathtub         |   1222255104 | 2008-09-24 06:18:24 |     4 |
| unniplucka      |   1222153815 | 2008-09-23 02:10:15 |     1 |
| totalfake       |   1214489830 | 2008-06-26 09:17:10 |     0 |
| IT_Dude         |   1210701456 | 2008-05-13 12:57:36 |     1 |
| cbthompson      |   1208207656 | 2008-04-14 16:14:16 |     0 |
| softweyr        |   1200931112 | 2008-01-21 09:58:32 |     2 |
| fuckyou         |   1174862102 | 2007-03-25 17:35:02 |     0 |
| shesatmine      |   1173185058 | 2007-03-06 06:44:18 |     8 |
| 2n00b4j00       |   1167362060 | 2006-12-28 21:14:20 |     2 |
| golfhakker      |   1143651653 | 2006-03-29 11:00:53 |     1 |
| konquistador    |   1141229063 | 2006-03-01 10:04:23 |     3 |
| john413         |   1128569081 | 2005-10-05 22:24:41 |     1 |
| kakra           |   1128167658 | 2005-10-01 06:54:18 |     0 |
| qwert           |   1121653682 | 2005-07-17 21:28:02 |     0 |
| reni            |   1112166778 | 2005-03-30 01:12:58 |     0 |
| pageld          |   1083773770 | 2004-05-05 11:16:10 |     4 |
| HipCracka       |   1070594834 | 2003-12-04 21:27:14 |     1 |
+-----------------+--------------+---------------------+-------+
115 rows in set (0.12 sec)

That (above) will preserve 309 posts during the next run (test):

mysql> select sum(posts) from user where email like '%123456789ABCDE%' ;
+------------+
| sum(posts) |
+------------+
|        309 |
+------------+
1 row in set (0.11 sec)

OBTW. User avatars have migrated over "OK" in this test round 2.

But the bbcode to markdown plugin did not work...... so need to fix that after this test run.

I check the migration of all vB forums and sub (child) forums to discourse categories.

This works fine; and it will only takes a few hours of time, once done, to reorganize the categories to make certain categories the parent or child category of another.

I think I'll ask one of our mods to volunteer for that task.

Anyone interested to do this, once I get 'final port" done?

Also, confirmed..... bbcode tags to markdown plugin is not working. Will have to fix this before the next test run.

still migrating....

importing posts...
   114156 / 651550 ( 17.5%)

I think I found the problem. In all the install directions and migration directions I read online (many) none of them mentioned setting the ENV["IMPORT"] variable

However, now that I am moving pass "complete ruby-noobie" stage, I looked at the Gemfile in discourse; and here is the issue why the bbcode to markdown plugin was not done:

if ENV["IMPORT"] == "1"
  gem 'mysql2'
  gem 'redcarpet'

  # NOTE: in import mode the version of sqlite can matter a lot, so we stick it to a specific one
  gem 'sqlite3', '~> 1.3', '>= 1.3.13'
  gem 'ruby-bbcode-to-md', git: 'https://github.com/nlalonde/ruby-bbcode-to-md'
  gem 'reverse_markdown'
  gem 'tiny_tds'
  gem 'csv'
end

So, I checked:

ubuntu-docker# env | grep -i IMPORT
ubuntu-docker# 

Nothing.

So now I know..... LOL... next build:

export IMPORT=1

So simple when you know the problem.

Hmmm

I'm convinced now. The posts and cautions on the net about Discourse is true. Yes, it is free and open source; and yes it is built on the model of selling services (consulting, hosting, custom programming) around the Discourse ecosystem.

Again, as in an earlier post, I'm not saying this is "good" or "bad"; but I'm staying "it is what it is"..... because, for example, this one small detail export IMPORT=1 was missing from every example, README and tutorial on the net. I'm sure those Discourse paid installers know about export IMPORT=1 ....

But then when we know the issue and when I search for "IMPORT=1 discourse" on the net, now that I know the problem and how to fit it, sure enough, there is it in a post out there:

Ruby mysql2 error on migration - support - Discourse Meta

Now we know..... LOL

I agree with the poster above. It would be nice if this small detail was made available to use at the beginning of the migration process; but hey, never mind... it's very nice free software and I figured it out on my own.

So, the next time I build this discourse app, it's export IMPORT=1 before building

Also:

MaxMind IP database updates require a license
Please set DISCOURSE_MAXMIND_LICENSE_KEY to one you generated at https://www.maxmind.com
MaxMind IP database updates require a license

Armed with this new knowledge ( export IMPORT=1 ), I have killed "test run 2" and have rebuilt for "test run 2b" .... and will start over with a new DB dump (but will not clear the discourse postgres db yet)

   239012 / 240522 ( 99.4%)
importing posts...
   177893 / 651614 ( 27.3%)   

Let's see what "test round 2b" looks like before moving on to "test round 3".....

PS: I still don't see the bbcode to markdown plugin in the admin panel but it definitely pulled it during the build:

Using rqrcode_core 0.1.1
Using rqrcode 1.1.2
Using rtlit 0.0.5
Using ruby-bbcode-to-md 0.0.14 from https://github.com/nlalonde/ruby-bbcode-to-md (at master@3b9aaf2)
Using ruby-readability 0.7.0
Using rubyzip 2.2.0

Starting to see, slowing but surely, this is gonna work and it's gonna be great.

This, when finished, will be a huge improvement for all users and the forums will be much better for all.

I am guessing the new discourse forum will be ready for member testing in less than a week from now.

importing posts...
   204459 / 651614 ( 31.4%) 

Migration notes, tentative plans, ideas:

• When I finishing migrating and testing and have a working discourse forum for the site I will open up the forum to all to visit and test.
• I will not turn off the main forums anytime soon and have no plans to turn off or change www.unix.com .
• The new forum will reside at discourse.unix.com.
• We will keep both sites for the time being, maybe indefinitely.
• If the testing goes well, and everyone is happy, I will set www.unix.com to read-only (reference) until sometime into the future.
• If the testing goes well, and everyone is happy, new posts will be at the new discourse site.

More to come.... this is going to work, based on what I am seeing.

I may bring this on line for testing sooner than later:

importing posts...
   264756 / 651614 ( 40.6%)

Just did a first quick cut at setting up categories based on legacy forums.

We will grant more people edit and mod privs based on more flexible discourse trust model.

Many "moderator emeritus" will have a higher trust level and will be able to edit posts in discourse.

This means we will greatly expand our leadership team based on trust levels, and many of our members who are active will be given a high trust level, by default when the new system goes live.

Edit:

importing posts...
   300396 / 651614 ( 46.1%)