Curious About the Tests

Are these the first times they've actually tried to port the old to the new?  I was under the impression that this conversion had already been done quite some time ago as a test, and users could continue "building" there on old threads from this forum.

If so, what has changed to cause the "real" conversions to fail?

Or am I totally misunderstanding what has actually taken place prior to last weekend?

Yes, it's odd

Yes, we did two prior test conversions. The first test conversion in March was surprisingly good for the very first one. After fixing the bugs we found, the second conversion in April was almost perfect, which is why after some testing we decided to do a live conversion.

However, the delay between April and September was 5 months (a long time) due to us getting covid and then being way behind and needing to catch up for several months. Once we finally got caught up, we decided to finally do the real go live.

Much to our surprise, this latest conversion was even worse than the first test conversion back in March.

No idea. Since the developers are in India (read: very affordable) my guess is turnover since April (5 months) meant the latest conversion attempt was by new developers and the previous conversion-savvy developers have moved on.

As long as we stay on top of the test conversions for the next few weeks, then we should be able to build a new cadre of conversion-experienced developers and finally get there.

Point of failure

I hope the conversion attempts are flagging what they're choking on... it might be as simple as deleting a single post to make it all work. On the other hand, it could be "posts containing the letter 'R'"... in which case "we'ye scyewed"

Post sequencing bug

I've been doing further research into the post sequencing bug and it appears to be a Drupal bug.

It looks like for whatever reason, folks who post from overseas can get original posting times that are BEFORE the posting times of other previous posts. Somehow, Drupal puts the post later in the thread even though the post time is earlier than several other posts.

If we go in and manually update the Authored on time to be later than the earlier posts instead of before them, then it should convert over to the new forum in the proper order.

Many posts by the prolific Australian contingent are out of order as to the Authored on timestamp, sometimes by a good half-dozen posts or even more. The only real solid fix appears to be editing the Authored on time to fit the post in between the other posts as to their timestamps.

Editing a post does NOT change the authored on timestamp, so that's no help. It appears the original authored on timestamp is bad from Drupal compared to the other posts around it. Drupal somehow still displays the posts in the "proper" sequence regardless of what the timestamp actually says.

Is Drupal using the comment # instead?

Instead of using the date/time, perhaps Drupal realized that wasn't going to work due to time zones and they organized the threads by comment number instead? 

Time

Timestamps work just fine for ordering, as long as ALL of them are in UTC.  It’s not uncommon for developers who aren’t used to thinking about the problem to mess up timezone stuff even when there are easy utilities to help deal with it.

Joe, you might be able to get a semi-automated fix going by guessing the timezone of the poster and correcting that to UTC, or whatever timezone the software is using for reference.  Might be easier to do as part of the conversion rather than fixing up beforehand in drupal?

If the original comment #

If the original comment # does not change when edited

Hi Jeff, Joef,

Seeing I was the one who detonated this landmine - The miss-ordering still happened if a post was edited days later. There was a distinct break of several days between when I did one of the mass edits on My Corrimal thread and the next big run I did to put topic headings into each of my posts before the April upload. There were several posts made between when the first run of edits and when the second run of edits were done. These "intermediate posts" by others that were not edited caused a distinct break in how the edited posts were loaded in the April upload.   

I can resend the previous Excel file I created if necessary. It gave before and after post #'s It only existed in for the April upload. I'm not sure how useful it would be for your programmers but it should point to individual postings that can be more fully investigated to see if they loaded correctly.   

If the posting # is not changed when we edit a post on this forum, then we have a unique # that doesn't change - unlike the upload/posting time. If the next upload can be done by increasing post # within each thread then we may have a way forward.   

Jeff may be onto something,  

Quick and dirty test - After posting this post was #487849 before editing. After posting this line and saving. 

Report back - response from system  https://forum.mrhmag.com/post/conversion-retry-also-failed-12219349  Post #487849 remains unchanged as the post #. 

More advanced test suggestion - Joe/Jeff post 1 comment only, and I'll edit this post again to see if I keep the post #. The timestamps on this post should show multiple times (or only the last one??)

Posting from "the Future...

As a quote I heard recently on YT said...

"...for those of you posting from in-the-future,

Or as some call it, Australia..."

Happy Modelling,

Aim to Improve,

Prof Klyzlr

Post number vs timestamp

Yes, no doubt, Drupal orders by post number, not timestamp. But unfortunately, the new forum orders by timestamp, and there’s the rub.

The easy fix would be to use the Drupal comment id number to sequence posts, but the new forum has no such ordering feature. To make the new forum use something other than timestamp ordering, they would have to redesign the forum software from the ground up, which ain’t going to happen.

As for setting all post times to UTC, that’s not going to help at all. The foreign posts are not off by hours, they’re only off by a few minutes, so it’s some time delay thing going on, not a time zone problem. Drupal appears to assign a post number as you start a post and adds a post timestamp when you save the post.

As a result, especially for foreign posts, between when the post was started and when it was saved, there can be a several minute delay. Meanwhile several quick posts get made by others that have later comment ID numbers but have earlier timestamps.

It gets even more interesting if the poster has more than one tab open and is working on answers to more than one comment in the same thread. The result can be posts with a Drupal sequence number that’s way out of sync with the timestamp. You can have two posts by the same person where an earlier post as to sequence number has a much later timestamp, putting it much further down the conversation stack.

It can get quite confusing because you can have a post that’s commenting on another post but the post they’re referencing is now actually several posts BELOW it. The thought process gets really jumbled as a result when it looks like someone is commenting on a post that “hasn’t happened yet.”

P.S. Here's a thought. When loading the data, they process the posts in Drupal ID order. The time of day is the problem, not the date. I wonder if they could look up the timestamp from the previous post in the thread and if it’s greater than the current post, just increment make the new post's timestamp by the previous post's timestamp plus one.

That way as they add posts they will make sure the next post they add always has a timestamp greater than the previous post.

Otherwise if the next post's timestamp is greater, then just use it. 

The result of this will be a slightly inaccurate timestamp for out of order posts, but the post sequence will still be correct, which I think is far more important.

The other thing we might do is run a timestamp repair process on the current forum's Drupal database. Essentially do this process and update the out of sync timestamp using this method.

Nasty

If I understand your description right Joe that’s a nasty one!  If the post was never edited after it’s original submit you might be able to use the mod date instead of the post date to get around it, but If it was edited later that would just make things worse.

If you’ve got someone who can still write some code on the drupal side, you could go through the posts in post # order looking for dates that were out of sequence, and fudge them into sequence.  It might take a little thought to get a good approach to the fudging that would work in the presence of some of the messier cases… perhaps a first pass through just to find runs of posts where the post # and dates agree on order, then you have start and end dates for the gaps you need to squeeze the out of order posts into by dividing the time interval equally to assign new dates.

Yes, I’m thinking we need to repair it ourselves

out of sequence timestamps ?

 Can you just let the timestamps fall as they may and move on? I doubt that many folks care what the order of past posts was. It's not too hard to read a thread and figure out the proper sequence ourselves and if it's still not clear what's the real harm?  I hate to see you have to spend extra money to fix something that doesn't give us a big benefit as readers? .....DaveB

Keeping correct post sequencing

Dear Dave,

If the (14?)-odd years worth of Forum postings were considered worthless in terms of collected knowledge and info, then I'd agree with you...

(Indeed, if they are genuinely of no-value, just dump the lot, hit the Big Reset, and start-again from scratch... it's not like the vast majority of forum readers use the Search Box to take self-serve advantage of said Freely-Available Knowledge Archive, based on the frequency of Oft-Repeated-Questions...   )

...however, if they _are_ considered valuable,

(Whixh I do honestly personally believe they are!),

then preserving "context sequence" and "follow the conversation" readability is absolutely important, esp fir someone who stumbles/searches/finds the threads days/months/years after the initial conversation, and have nothing but the zero-context text-on-screen to orient themselves with...

Personally, more than happy to give Joe and the Coders (sounds like a 90s proto-techno band) all the time required to "get it sorted"...

Happy Modelling,

Aim to Improve,

Prof Klyzlr

Man, that explains a lot of

Man, that explains a lot of quirkiness I've seen when leaving an unfinished post opened for a long time and being utterly confused where it landed in the thread when I get back. 

< internal screaming at Drupal>

It causes real harm

It causes real harm. For example, in the DCC command station thread, Prof's often commenting on posts that come several posts after him (and not before him like they should) and it's really confusing to read. Looks really awful.

I just did a test of how many timestamps are whacked, and its about 6700 posts.

I also tested an automated repair process and it works, but it's going to take me a day or so to "get it sorted" as they say. I'm just taking an out-of-sequence timestamp and adding one second to the previous post timestamp and updating the out-of-sequence post to put it where it goes.

  "Personally, more than

Hi Prof,  I certainly don't care if it takes them longer, I just hate to see him spend more money fixing something that wouldn't bother me much. If all the info is there i can easily unscramble the sequence while reading the thread ....DaveB

6700 threads and comments on

6700 threads and comments on posts that appear after the comment is pretty hard to handle. 

Trust me

Trust me, you will notice. It makes some threads totally confusing -- the recent DCC Command Station thread is a total confusing mess with all kinds of posts out of order throughout. You start reading it and you go .... "Wha--- ???" and then you compare it to the thread over here and you go ... dang, this converted thread is all jumbled up.

No extra money, just a bit of time on my part, since I'm fixing the Drupal database myself. I was a Database Admin in my day job for many years before I became a manager -- and even then I often had DBA discussions with my staff where we got down into the coding weeds.

Takes hint...

(Prof takes hint, invokes self-imposed posting limit)

No worries

No worries, I figure just before we do the next go live attempt, I will scan all posts since my timestamp patch date and we will run the patch one more time before I hand off the database to the developers.

I'm actually relieved I finally figured it out. It will all be fixed (for now) in a day or so. It was actually kinda fun figuring out an automated solution and to find out I still have some database chops left.

Readability suffers BIG Time

Dear Dave, all,

Readability suffers BIG time when an upload goes sideways like this probable worst case from the April upload - in just over 400 posts:

what was post #5 went to position #194

what was post #8 went to position #149

what was post #9 went to position #150

what was post #11 went to position #151

what was post #14 went to position #153

what was post #48 went to position #175

what was post #49 went to position #209

what was post #50 went to position #176

what was post #51 went to position #177

what was post #58 went to position #184

what was post #60 went to position #185

That's a very incomplete list with 64 errors in the first 100 of those 400+ posts in the topic when uploaded in April. 

Good luck trying to make any sort of coherent sense out of results like that.  Now just suppose it was a topic containing a sequential build of someone else's layout or project?  Oh, it wouldn't matter??  I don't think so and neither would they.

It took me just on 3 days work to manually track what went from where it was supposed to be in the current forum to "somewhere else" where it actually landed after that April upload for just the first 100 entries or 25% of that topic. I stopped trying to work out what went where after those matching the first 100 posts on this current forum. Another fix was suggested and tried before the last upload. I don't know if this fix will work when the next upload occurs. But what is being uploaded this time for this topic will be considerably different in timestamps to what was uploaded in April.

I'm glad Joe found an automated way to do the fix. My manual workaround for 400+ posts took days to do. I just hope Joe's time fix works. Because the fixers are in for a lot of work before or after the next upload if it doesn't, with possibly another 6000+ posts in ??? topics affected, not just the topic I've documented above. 

Timestamp "fix" doesn't work

YAAY!!!!

Thanks Joe,

YAAY!!! Your database efforts are truly appreciated by myself. I can see how this will make the next upload go so much better. 

Many thanks,  

Is it possible that members

Is it possible that members could "update" their profile in such a way that, for example, I would be browsing the forums in (is it ? )  eastern pacific time -or whichever time zone you are actually in? - therefore any postings I make would, as far as the forum software is concerned, be posted in "local time" ?

 I'm a complete computer numpty so this is probably drivel or not possible but is a suggestion?

Keith  (In France)