VBulletin 3.8 to Discourse on Docker Migration Test Take Four

Test Build 4 on New Server, with changes identified in discourse test builds 2 and 3, primarily:

  • Insuring ruby-bbcode-to-markdown is enabled.
  • Removing line breaks from ICODE to markdown in migration script.
  • Added vbpostid to posts in discourse to setup migrating vb "thanks" to discourse "likes" (not sure if this part will work or not, first time test for this migration feature.)

Chuggin' along again, from total scratch on a new server (and with fingers crossed):

root@discourse1-app:/var/www/discourse# su discourse -c 'bundle exec ruby script/import_scripts/vbulletin_neo4.rb'
discourse:@localhost wants vb3
Loading existing groups...
Loading existing users...
Loading existing categories...
Loading existing posts...
Loading existing topics...

importing groups..
       20 / 20 (100.0%)  [173261 items/min]   
importing users
     4774 / 138160 (  3.5%)  [182 items/min] 

Looks like the "user part" of the migration will take about 12 hours :frowning:

Entry in discourse posts table , to set up "thanks" to "likes" migration:

 vbpostid                | integer                     |           |          | 

Let's see if this new vbulletin_neo4.rb code works on the first attempt (if it does, that would be a pleasant surprise).

Note: Will not be migrating private messages (PMs) from vb to discourse in this test run and probably not in the final migration either.

Moving along...

importing users
    62848 / 138160 ( 45.5%)  [181 items/min] 

We have two servers, one for MIGRATION TEST 4 (above, current test) and a future site staged and ready for the "final migration" if when the migration and testing is done.

Don't want to get too excited until we see the results of T4, and if the ICODE and BBCODE tags migrate well this go around; and if the vB "Thanks" get staged so we can transfer all vB "Thanks" to "Likes"

It's not a matter of "if" but a matter of "when" at this point, from what I see.

It will be "quite an accomplishment" to migrate a long EOL vB to a modern, sleek, community-forum like Discourse. When I started the eval a few days ago, the very talented folks at Discourse said "you are on your own on this one, pal" (due to the fact we are on vB3) and closed my vB3 bug report with a "good luck" finale. It's too early to say "amazing work"... but it's getting close. It's actually very good at the end of T3, but I am "going for even better".

1 Like

Moving right along..... no apparent errors in the modifications included in the Ruby migration script, vbulletin_neo4.rb :

importing groups..
       20 / 20 (100.0%)  [49738 items/min]    
importing users
   138160 / 138160 (100.0%)  [407 items/min]       ]  
Creating groups membership...
    Done
importing all forums  top level categories...
      119 / 119 (100.0%)  [423 items/min] 
importing topics...
   171065 / 240534 ( 71.1%)
2 Likes

Import is going well. My custom code for our ICODE tag and BBCODE to markdown is working fine. Everything is looking good:

... still moving long now importing posts (replies):

importing topics
   236002 / 240534 ( 98.1%)  [185691 items/min]  
importing posts...
     8597 / 651677 (  1.3%) 

Did not import private messages (PMs).

Will attempt to import all attachments, but if they do not import nicely, it not a show stopper; and it will save disk and file space not importing images and files from many years ago;

The person who posts the most images as attachments is me, and I am OK if all of the do not import / migration perfectly.

The most important thing is the code, and all CODE tags and ICODE tags will migration to markdown with issue.

The only "gotcha" so far is the that it appears the vbpostid for importing the "thanks" to "likes" is not working as planned, so I will need to think how to fix this after this test run is done.

discourse=> select vbpostid from posts order by vbpostid asc limit 10;
 vbpostid 
----------
         
         
         
         
         
         
         
         
         
         
(10 rows)
discourse=> select vbpostid from posts where vbpostid > 0 order by vbpostid asc limit 10;
 vbpostid 
----------
(0 rows)

Time for some Ruby puts statements in the script to see where the bug is.

1 Like

puts shows we are getting

vbpostid

from the mysql db ok, but it's not writing to the postgres db.

My best guest was that the structure of the posts table for discourse is defined somewhere in a lib I'm was not familiar with yet.

So, I searched and found this directory:

/var/www/discourse/script/import_scripts/base

and this script:

generic_database.rb

and added vbpostid to this code:

But still no joy....

I have a solution to this issue, or at least a workaround:

EDIT: This method only works on posts with no bbcode because discourse preprocesses the raw posts and changes the post. Hence, the MD4 (or matching methods in general, do not work).

raw = preprocess_post_raw()

On discourse (after migrating):

discourse=> select md5(raw) from posts where raw like '%where can i get a free copy of unix? any kind... plz send %';
               md5                
----------------------------------
 e136704539982b094d8d1d4229884583
(1 row)

On vb forums:

mysql> select md5(pagetext) from post where pagetext like '%where can i get a free copy of unix? any kind... plz send %';
+----------------------------------+
| md5(pagetext)                    |
+----------------------------------+
| e136704539982b094d8d1d4229884583 |
+----------------------------------+
1 row in set (1.07 sec)


So, the MD5() function in post mysql and postgres returns the same hash on the same post in two different DBs (it's should, but always good to check).

This means that we can match the posts in both DBs without unique post ids using MD5() and the user name who created the post.

The

MD5

hash combined with the user name should does not work because the post is preprocessed for bbcode and markdown .

Hmmmm does not work consistently.... must be converting some chars in the pagetext of a vb post to postgres.

Update: Yes, the posts are preprocessed before migration for markdown and bbcode, so MD5 will not work.

discourse=> select id,md5(raw) from posts where raw like '%Is it possible to see the mails received by him (mail sent to him) in outlook on my Windows workstation?%';
   id   |               md5                
--------+----------------------------------
 377402 | 04d0a536ccd72d8523309b0a4c5da55d
(1 row)
mysql> select postid,md5(pagetext) from post where pagetext like '%Is it possible to see the mails received by him (mail sent to him) in outlook on my Windows workstation?%';
+-----------+----------------------------------+
| postid    | md5(pagetext)                    |
+-----------+----------------------------------+
| 303045132 | 61f0d1958e9002b69872bebb88a499b9 |
+-----------+----------------------------------+
1 row in set (0.99 sec)

Darn it.

Still moving along.....

importing posts...
   548360 / 651677 ( 84.1%) 

The only issues I see are related to:

  • Migrating "vb thanks" to "discourse likes" (have not found a reliable simple solution to this yet except, of course pattern matching in the postgres DB is always an option, but this will be painfully slow).
  • Not sure how all the various kinds of attachments will migrate, since the vb site has many different legacy attachment methods, so we may not migrate over most of them.

Will not migrate the following:

  • PMs (no reason to migrate these).
  • Passwords (the hash functions are different so better to reset using "lost password", which works fine and all password will follow the discourse rules),

I have another potential solution to this "thanks to likes" (vb to discourse) issue:

This idea did not work. - The discourse postids do not match as expected.

Before the ruby migration create_post loop:

   neo_post_count = DB.exec <<~SQL
          SELECT count(id) from posts
   SQL

Inside the migration create_post loop:

neo_post_count += 1
mysql_query("UPDATE user_actions SET discourse_post_id = #{neo_post_count} WHERE target_post_id = #{pid} AND user_id = #{tuid}")

Update: Did not work. I'm not really understanding ruby, the discourse db libs and postgres yet.

Still problems with attachment migration:

importing attachments...
   269043 / 651677 ( 41.3%)  Couldn't find attachment record for post.id = 269043, import_id = thread-139764
   323104 / 651677 ( 49.6%)  Couldn't find attachment record for post.id = 323104, import_id = thread-213737
   354790 / 651677 ( 54.4%)  Couldn't find attachment record for post.id = 354793, import_id = thread-260244
   375104 / 651677 ( 57.6%)  Traceback (most recent call last):
	23: from script/import_scripts/vbulletin_neo4.rb:968:in `<main>'
	22: from /var/www/discourse/script/import_scripts/base.rb:47:in `perform'
	21: from script/import_scripts/vbulletin_neo4.rb:97:in `execute'
	20: from script/import_scripts/vbulletin_neo4.rb:602:in `import_attachments'
	19: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/querying.rb:21:in `find_each'
	18: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:69:in `find_each'
	17: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:135:in `find_in_batches'
	16: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:222:in `in_batches'
	15: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:222:in `loop'
	14: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:238:in `block in in_batches'
	13: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
	12: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:70:in `block in find_each'
	11: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:70:in `each'
	10: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
	 9: from script/import_scripts/vbulletin_neo4.rb:607:in `block in import_attachments'
	 8: from script/import_scripts/vbulletin_neo4.rb:607:in `gsub!'
	 7: from script/import_scripts/vbulletin_neo4.rb:617:in `block (2 levels) in import_attachments'
	 6: from /var/www/discourse/script/import_scripts/base.rb:847:in `html_for_upload'
	 5: from /var/www/discourse/script/import_scripts/base/uploader.rb:40:in `html_for_upload'
	 4: from /var/www/discourse/lib/upload_markdown.rb:10:in `to_markdown'
	 3: from /var/www/discourse/lib/upload_markdown.rb:17:in `image_markdown'
	 2: from /var/www/discourse/app/models/upload.rb:123:in `short_url'
	 1: from /var/www/discourse/app/models/upload.rb:397:in `short_url_basename'
/var/www/discourse/app/models/upload.rb:161:in `base62_sha1': undefined method `hex' for nil:NilClass (NoMethodError)

So, changed method hex in upload.rb to sha1 ... trying again.

Did not work either...

/var/www/discourse/app/models/upload.rb:161:in `base62_sha1': undefined method `sha1' for "a5019db7160084fb98dc8a9398bcad8ad05a42ce":String (NoMethodError)

Added ruby rescue for exception handling;

importing attachments...
   375104 / 651677 ( 57.6%)  skipping....base62.encode.... 
skipping....base62.encode.... 
skipping....base62.encode.... 
skipping....base62.encode.... 
   376880 / 651677 ( 57.8%)  Couldn't find attachment record for post.id = 376883, import_id = thread-283391
   704284 / 651677 (108.1%)  Couldn't find attachment record for post.id = 704340, import_id = 302453503
   764647 / 651677 (117.3%)  Couldn't find attachment record for post.id = 764703, import_id = 302538540
   792031 / 651677 (121.5%)  

Seems to be moving along again but then another error:

/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:126:in `call': LOADING Redis is loading the dataset in memory (Redis::CommandError)

Added another rescue to catch and skip this:

Lots of error, but it's moving on past attachment migration errors now:

  1026248 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026251 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026261 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026299 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026402 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026403 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026405 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026454 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026457 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026462 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026473 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026489 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026506 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026522 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026533 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026538 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026540 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026543 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026576 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026600 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026611 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026636 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026656 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026659 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026660 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026661 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026676 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026677 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026679 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026680 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026682 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026683 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026685 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026715 / 651677 (157.5%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026721 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026741 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026746 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026747 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026759 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026760 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026771 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026922 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026923 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026984 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1026988 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027046 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027063 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027071 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027073 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027082 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027089 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027114 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027115 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027149 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027172 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027204 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027211 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027349 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027364 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027366 / 651677 (157.6%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027371 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027445 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027512 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027571 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027583 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027584 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027585 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027645 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027648 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027649 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027650 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027660 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027708 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027715 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027759 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027764 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027768 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027803 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027816 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027817 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027824 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027842 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027914 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027934 / 651677 (157.7%)  Failed to create upload: undefined method `[]' for nil:NilClass
Upload not valid :(
  1027936 / 651677 (157.7%)  
Closing topics...

No one likes migrations....

LOL

Some of my crude hacking worked ..... LOL

At least some attachments are working:

Success! LOL

2 Likes

Found a ruby script in the import_scripts directory of the discourse app which I have modified to transfer vb3 plugin "thanks" to discourse "likes":

def import_likes
    puts "\nimporting likes..."

     # neo created this mysql user_actions table from vb3 and added it to the mysql dump for migration
    sql = "select user_id, target_post_id, created_at from user_actions"
    results = mysql_query(sql)

    puts "loading unique id map"
    existing_map = {}
    PostCustomField.where(name: 'import_unique_id').pluck(:post_id, :value).each do |post_id, import_id|
      existing_map[import_id] = post_id
    end

    puts "loading data into temp table"
    DB.exec("create temp table like_data(user_id int, post_id int, created_at timestamp without time zone)")
    PostAction.transaction do
      results.each do |result|

        result["user_id"] = user_id_from_imported_user_id(result["user_id"].to_s)
        result["post_id"] = existing_map[result["post_id"].to_s]

        next unless result["user_id"] && result["post_id"]

        DB.exec("INSERT INTO like_data VALUES (:user_id,:post_id,:created_at)",
          user_id: result["user_id"],
          post_id: result["post_id"],
          created_at: result["created_at"]
        )

      end
    end

    puts "creating missing post actions"
    DB.exec <<~SQL

    INSERT INTO post_actions (post_id, user_id, post_action_type_id, created_at, updated_at)
             SELECT l.post_id, l.user_id, 2, l.created_at, l.created_at FROM like_data l
             LEFT JOIN post_actions a ON a.post_id = l.post_id AND l.user_id = a.user_id AND a.post_action_type_id = 2
             WHERE a.id IS NULL
    SQL

    puts "creating missing user actions"
    DB.exec <<~SQL
    INSERT INTO user_actions (user_id, action_type, target_topic_id, target_post_id, acting_user_id, created_at, updated_at)
             SELECT pa.user_id, 1, p.topic_id, p.id, pa.user_id, pa.created_at, pa.created_at
             FROM post_actions pa
             JOIN posts p ON p.id = pa.post_id
             LEFT JOIN user_actions ua ON action_type = 1 AND ua.target_post_id = pa.post_id AND ua.user_id = pa.user_id

             WHERE ua.id IS NULL AND pa.post_action_type_id = 2
    SQL

    # reverse action
    DB.exec <<~SQL
    INSERT INTO user_actions (user_id, action_type, target_topic_id, target_post_id, acting_user_id, created_at, updated_at)
             SELECT p.user_id, 2, p.topic_id, p.id, pa.user_id, pa.created_at, pa.created_at
             FROM post_actions pa
             JOIN posts p ON p.id = pa.post_id
             LEFT JOIN user_actions ua ON action_type = 2 AND ua.target_post_id = pa.post_id AND
                ua.acting_user_id = pa.user_id AND ua.user_id = p.user_id

             WHERE ua.id IS NULL AND pa.post_action_type_id = 2
    SQL
    puts "updating like counts on posts"

    DB.exec <<~SQL
        UPDATE posts SET like_count = coalesce(cnt,0)
                  FROM (
        SELECT post_id, count(*) cnt
        FROM post_actions
        WHERE post_action_type_id = 2 AND deleted_at IS NULL
        GROUP BY post_id
    ) x
    WHERE posts.like_count <> x.cnt AND posts.id = x.post_id

    SQL

    puts "updating like counts on topics"

    DB.exec <<-SQL
      UPDATE topics SET like_count = coalesce(cnt,0)
      FROM (
        SELECT topic_id, sum(like_count) cnt
        FROM posts
        WHERE deleted_at IS NULL
        GROUP BY topic_id
      ) x
      WHERE topics.like_count <> x.cnt AND topics.id = x.topic_id

    SQL
  end

Will try this after the current migration test build 4 is done.....

It's still running.....

Closing topics...
1 Like

Moving along.....

Closing topics...
	``````````
Postprocessing posts...
   556410 / 1027936 ( 54.1%) 

Close to baked:

Postprocessing posts...
  1027939 / 1027939 (100.0%)  
Creating Permalink File...


updating banned users
     3134 / 3134 (100.0%)  

Updating topic status

Updating bumped_at on topics

Updating last posted at on users

Updating last seen at on users

Updating topic reply counts...
    12466 / 138032 (  9.0%)  [9565 items/min]   
discourse=> select id, name from groups;
discourse=> update users set primary_group_id = 2 where primary_group_id = 48;      #moved all moderators in vb to mod group on discourse

In addition:

  • Moved all moderators and moderator emeritus to trust_factor_4 (Leader)
  • Moved all forum advisors to trust_factor_3

Done...

Updating topic reply counts...
   138032 / 138032 (100.0%)  [10714 items/min]    
Updating first_post_created_at...

Updating user post_count...

Updating user topic_count...

Updating topic users

Updating post timings

Updating featured topic users

Updating featured topics in categories
      123 / 123 (100.0%)  [81 items/min]   n]  
Updating user topic reply counts
   138032 / 138032 (100.0%)  [11368 items/min]  ]  
Resetting topic counters


Done

Backing up now.....

Then will try to migrate "vb thanks" to "discourse likes" with the new script vbulletin_neo5.rb

3 Likes

All is done, for the initial migrations except for "vb thanks" to "discourse likes".

(of course the discourse site is behind the vb site as far as the most recent posts....)

It's looking good, but I would like to get the "vb thanks" to "discourse likes" working.

1 Like

Have been debugging ruby code this morning for 2 to 3 hours (more like 3) to migrate "thanks" to "likes" and I'm getting close.

I am hopeful I this get this script done today, and have a working script to transfer all our vb3 thanks plugin "thanks" to discourse "likes".....

This migration has been a task "not for the faint of heart" and there is no doubt anyone who tries this must have a lot of technical skills.

Just now, for example, I debugged a lot of ruby code were I was getting "trying to update DB errors, unique" and searched the code in ruby, found the tables and found the unique index. I deleted the "unique index" which allowed the process to move forward and then after it was complete, I could see the error in the results and was about to go back and make the change to the original DB user mappings.

root@discourse1-app:/var/www/discourse# rake posts:delete_all_likes   
    65028 / 65028 (100.0%)
65028 likes deleted!

Sometimes you have to break something to fix it.

I started this "feasibility study" caper around a week ago, and I'm getting close to finishing (the entire migration) ; that's not bad for a vB3 migration where the folks at meta discourse gave me the serious "get out of here" cold shoulder when they found out I was migrating a vB3 forum to discourse.