Update from 2.4.20 to 20220803.1 failed, two issues so far (Bugs)

by Nico Hoffmann @, (863 days ago)

Hello,

I tried the update like written in the subject and experienced two issues. Both issues break the update and leave the database in a state which is not working for the old version.

1. the update script printed out "Database error in line 452: Specified key was too long; max key length is 767 bytes" and the update did not finish.
I noticed that this issue was discussed already, and I also did some googeling. Probably this isuue is somehow related to an old SQL version, in my case it is "Server version: 5.5.62 MySQL Community Server (GPL)", i.e. I agree with the explanation from user Auge.

I did not really understand if there is already a bugfix. If yes, it looks like it did not work properly. Maybe the best solution is a SQL update.


2. just for fun i tried a quick-and-dirty hack (not recommended for others). I changed in the update script line 448 "VARCHAR(256)" to "VARCHAR(255)", because 3 * 255 < 768. And indeed, this error was gone, but another appeared: "Database error in line 452: Duplicate entry 'abcdef@domain.tld' for key 'key_user_email'". It turned out that there are two different usernames with the same email address.
Thats a case of a bad administration, in the first place :-) I will remove the double entry.

But I think it should checked before updating or at least mentioned in the update hints.


I installed the backup of my current forum, so there is no damage and I can prepare the next try without stress. While installing the database backup I noticed the new version added four tables. i.e. PREFIX-akismet_rating, PREFIX-b8_rating, PREFIX-b8_wordlist, and PREFIX-uploads.
It turned out that these tables remained in the database after installing the backup and caused errors at the next try.

I would suggest to offer some short fallback hints in the update description, maybe:
"Before update: back up database and php/html code.
If it did not work:
replace code with your backup
replace database with your backup
delete the newly added tables, if still present"

have a nice day!

Avatar

Update from 2.4.20 to 20220803.1 failed, two issues so far

by Auge ⌂, (862 days ago) @ Nico Hoffmann

Hello

Und ewig grüßt das Murmeltier … Groundhog Day again.

I tried the update like written in the subject and experienced two issues. Both issues break the update and leave the database in a state which is not working for the old version.

1. the update script printed out "Database error in line 452: Specified key was too long; max key length is 767 bytes" and the update did not finish.
I noticed that this issue was discussed already, and I also did some googeling. Probably this isuue is somehow related to an old SQL version, in my case it is "Server version: 5.5.62 MySQL Community Server (GPL)", i.e. I agree with the explanation from user Auge.

I did not really understand if there is already a bugfix. If yes, it looks like it did not work properly. Maybe the best solution is a SQL update.

Yes, I thought I fixed it. But when this problem encountered for you with the latest available version 20220803.1, it obviously doesn't work in every case.

2. just for fun i tried a quick-and-dirty hack (not recommended for others). I changed in the update script line 448 "VARCHAR(256)" to "VARCHAR(255)", because 3 * 255 < 768.

I will check, if this is an appropriate way, a.k.a. a not so quick-and-dirty hack as you think, to solve the issue. I solved the same issue with setting the charset to utf8 (without the appendix mb4). That way the maximum index length does not exceed because of the different charset. But it was obviously the wrong place to do this.

And indeed, this error was gone, but another appeared: "Database error in line 452: Duplicate entry 'abcdef@domain.tld' for key 'key_user_email'". It turned out that there are two different usernames with the same email address.
Thats a case of a bad administration, in the first place :-) I will remove the double entry.

I don't know, if this is really a case of a "bad administration". Until version 20220508.1 (2.5.0) it was simply possible to add more than one account with the same e-mail-address. I for myself used this possibility regulary for testing purposes.

It's a trap when updating to a version in the 2.5 branch.

But I think it should checked before updating or at least mentioned in the update hints.

Yes, you are right.

Once we changed the collation of the column username in the userdata table because the database found the name "Hans" and the fictive name "Häns" to be identical. The update script tested the database for such pseudoidentical values, to enforce the admin to solve the problem in collaboration with the affected users before the update.

This time (making the e-mail-address unique) we didn't checked the values for doublettes. That is a failure. I will have to completely rework the update script.

I installed the backup of my current forum, so there is no damage and I can prepare the next try without stress. While installing the database backup I noticed the new version added four tables. i.e. PREFIX-akismet_rating, PREFIX-b8_rating, PREFIX-b8_wordlist, and PREFIX-uploads.
It turned out that these tables remained in the database after installing the backup and caused errors at the next try.

Hmm. The update script doesn't check for already existing tables. Normally this will not happen, but in your case with a previoulsy partially failed upgrade, it will fail.

I would suggest to offer some short fallback hints in the update description, maybe:
"Before update: back up database and php/html code.
If it did not work:
replace code with your backup
replace database with your backup
delete the newly added tables, if still present"

ACK

Tschö, Auge

--
Trenne niemals Müll, denn er hat nur eine Silbe!

Update from 2.4.20 to 20220803.1 failed, two issues so far

by Nico Hoffmann @, (848 days ago) @ Nico Hoffmann

Well, the next try :-)

my provider offered a transfer to a new database with MYSQL 5.7 (also possible MYSQL 8, or MariaDB 10.6) and I tried the update. I got these errors:


Fehler!

    Database error in line 363: Incorrect datetime value: '0000-00-00 00:00:00' for column 'edited' at row 1
    Database error in line 378: Incorrect date value: '0000-00-00' for column 'birthday' at row 1
    Database error in line 381: Incorrect date value: '0000-00-00' for column 'birthday' at row 1

The forum itself seems to run, however it shown ugly PHP warnings:


Warning: mysqli_fetch_row() expects parameter 1 to be mysqli_result, boolean given in /PATH/includes/main.inc.php on line 259

Warning: mysqli_free_result() expects parameter 1 to be mysqli_result, boolean given in /PATH/forum/includes/main.inc.php on line 260

Warning: mysqli_fetch_row() expects parameter 1 to be mysqli_result, boolean given in /PATH/forum/includes/main.inc.php on line 262

Warning: mysqli_free_result() expects parameter 1 to be mysqli_result, boolean given in /PATH/forum/includes/main.inc.php on line 263

Warning: mysqli_fetch_row() expects parameter 1 to be mysqli_result, boolean given in /PATH/forum/includes/main.inc.php on line 267

Warning: mysqli_free_result() expects parameter 1 to be mysqli_result, boolean given in /PATH/forum/includes/main.inc.php on line 268

Warning: Cannot modify header information - headers already sent by (output started at /PATH/forum/includes/main.inc.php:259) in /PATH/forum/index.php on line 208

Warning: Cannot modify header information - headers already sent by (output started at /PATH/forum/includes/main.inc.php:259) in /PATH/forum/index.php on line 209

remarks:

- Before the update attempt today the forum was O.K., but I did a similar try two weeks ago. I cannot confirm that this try did not touch the databases. However this should be unlikely.

- In the "entries" table there are entries with "0000-00-00 00:00:00" in the edited row.

- in the "uderdata" table there are entries with "0000-00-00" in the borthday row.

Should I set these entries to NULL manually before updating?

Any suggestions?

Avatar

Update from 2.4.20 to 20220803.1 failed, two issues so far

by Auge ⌂, (848 days ago) @ Nico Hoffmann

Hello

Well, the next try :-)

As first: Thank you for your unintentionally tests.

my provider offered a transfer to a new database with MYSQL 5.7 (also possible MYSQL 8, or MariaDB 10.6) and I tried the update. I got these errors:


Fehler!

Database error in line 363: Incorrect datetime value: '0000-00-00 00:00:00' for column 'edited' at row 1
Database error in line 378: Incorrect date value: '0000-00-00' for column 'birthday' at row 1
Database error in line 381: Incorrect date value: '0000-00-00' for column 'birthday' at row 1

That's the pitfalls of the configuration dependent, possible lax data model of MySQL and the default values for columns that was mostly set by Alex years ago (possibly wíthout the necessary knowledge about the corner cases, i.e. much more stringent configurations of MySQL-servers).

The forum itself seems to run, however it shown ugly PHP warnings:


Warning: mysqli_fetch_row() expects parameter 1 to be mysqli_result, boolean given in /PATH/includes/main.inc.php on line 259

Warning: mysqli_free_result() expects parameter 1 to be mysqli_result, boolean given in /PATH/forum/includes/main.inc.php on line 260

Warning: mysqli_fetch_row() expects parameter 1 to be mysqli_result, boolean given in /PATH/forum/includes/main.inc.php on line 262

Warning: mysqli_free_result() expects parameter 1 to be mysqli_result, boolean given in /PATH/forum/includes/main.inc.php on line 263

Warning: mysqli_fetch_row() expects parameter 1 to be mysqli_result, boolean given in /PATH/forum/includes/main.inc.php on line 267

Warning: mysqli_free_result() expects parameter 1 to be mysqli_result, boolean given in /PATH/forum/includes/main.inc.php on line 268

Hmmm, the error in line 259 is the cause of the following errors. This is the complete block in main.inc.php (version 20220803.1).


259 | $count_result = mysqli_query($connid, "SELECT COUNT(*) FROM " . $db_settings['userdata_table'] . " WHERE activate_code = ''");
260 | list($registered_users) = mysqli_fetch_row($count_result);
261 |
262 | if ($settings['count_users_online'] > 0) {
263 |  user_online($settings['count_users_online']);
264 |  $count_result = mysqli_query($connid, "SELECT COUNT(*) FROM ".$db_settings['useronline_table']." WHERE user_id > 0");
265 |  list($registered_users_online) = mysqli_fetch_row($count_result);
266 |  $count_result = mysqli_query($connid, "SELECT COUNT(*) FROM ".$db_settings['useronline_table']." WHERE user_id = 0");
267 |  list($unregistered_users_online) = mysqli_fetch_row($count_result);
268 |  $total_users_online = $unregistered_users_online + $registered_users_online;
269 | }
270 | mysqli_free_result($count_result);

But comparing the line numbers in your error messages with the code one can see, that the messages reports MySQL-errors in lines, where we don't see MySQL-related function calls in the code of version 20220803.1 (as shown above). But when we look into the code of version 2.4.20 (the version from where you started the update) we can see MySQL-related code with a query, that includes columns that are gone with version 2.5. The gone colum is part of the string in the variable $display_spam_query_and, that was generated in the previous block, beginning with line #239.


256 | } else {
257 |  // there are categories
258 |  $count_result = mysqli_query($connid, "SELECT COUNT(*) FROM ".$db_settings['forum_table']." WHERE pid = 0".$display_spam_query_and." AND category IN (".$category_ids_query.")");
259 |  list($total_threads) = mysqli_fetch_row($count_result);
260 |  mysqli_free_result($count_result);
261 |  $count_result = mysqli_query($connid, "SELECT COUNT(*) FROM ".$db_settings['forum_table']." WHERE category IN (".$category_ids_query.")".$display_spam_query_and);
262 |  list($total_postings) = mysqli_fetch_row($count_result);
263 |  mysqli_free_result($count_result);
264 | }
265 | // count spam:
266 | $count_spam_result = mysqli_query($connid, "SELECT COUNT(*) FROM ".$db_settings['forum_table']." WHERE spam = 1");
267 | list($total_spam) = mysqli_fetch_row($count_spam_result);
268 | mysqli_free_result($count_spam_result);

That said, we have to assume, that you forgot to upload the script file(s) (at least the main.inc.php).


Warning: Cannot modify header information - headers already sent by (output started at /PATH/forum/includes/main.inc.php:259) in /PATH/forum/index.php on line 208

Warning: Cannot modify header information - headers already sent by (output started at /PATH/forum/includes/main.inc.php:259) in /PATH/forum/index.php on line 209

These are subsequent errors because of the output of the previous error messages. This errors would not occur, if the error in main.inc.php, line 259 would not happen.

remarks:

- Before the update attempt today the forum was O.K., but I did a similar try two weeks ago. I cannot confirm that this try did not touch the databases. However this should be unlikely.
- In the "entries" table there are entries with "0000-00-00 00:00:00" in the edited row.
- in the "uderdata" table there are entries with "0000-00-00" in the borthday row.

Should I set these entries to NULL manually before updating?

Generally this is a proper idea, but I cannot recommend this without knowing the possible side effects (i.e. when a function relies on the format (0000-00-00 (00:00:00)) and does not take a NULL value into account). There are many more places in the database, where setting the default values to NULL would make IMHO much more sense than the current defaults does.

Any suggestions?

I had a few looks into the code of the update script and I have ideas to improve the script without breaking updates, that have aready been made. I will inspect the code with a focus on the date and datetime topics today and will give a report today in the evening or tomorrow.

Tschö, Auge

--
Trenne niemals Müll, denn er hat nur eine Silbe!

Update from 2.4.20 to 20220803.1 failed, two issues so far

by Nico Hoffmann, (800 days ago) @ Auge

my provider offered a transfer to a new database with MYSQL 5.7 (also possible MYSQL 8, or MariaDB 10.6) and I tried the update. I got these errors:


Fehler!

Database error in line 363: Incorrect datetime value: '0000-00-00 00:00:00' for column 'edited' at row 1
Database error in line 378: Incorrect date value: '0000-00-00' for column 'birthday' at row 1
Database error in line 381: Incorrect date value: '0000-00-00' for column 'birthday' at row 1

Well, I can give an update to this issues.

This is what I did:

- first I made a backup from the database and the php stuff. And indeed, at some step in the following trial-and-error-game I made a mistake and corrupted the database. So I needed the SQL dump!

- since I had already some difficulties, I duplicated both, i.e. the database and the web files. One instance for testing some changes at the database and one instance for porting back in case of success.

- technically spoken, the main point was to obtain a database which is updatable. In my case, I found three obstacles (after migrating from MYSQL 5.6 to MYSQL7 or higher):

-- first: the table PREFIX_userdata contains email the address for every account. In the past, it was no problem to have two different accounts with the same email address. The update script however does not accept this (as I mentioned already somewhere), so this table must contain every email address only once. Google pointed me to this MYSQL snipplet, which lists all email addresses with more than one occurrence:

mysql> select
-> user_email,
-> count(user_email)
-> From
-> PREFIX_userdata
-> group by (user_email)
-> having count(user_email) > 1;

Then, the "double" accounts must removed.

-- second: if the update script does finish with with errors, it may already have changed the database. Especially, it creates some new tables. Loading the backup SQL dump does not remove the newly created tables, but a second run of the update script leads to an error, because it relys on a clean database. Therefore, the newly created tables must be dropped. To my knowledge this are four tables. "drop table PREFIX_akismet_rating,PREFIX_b8_rating,PREFIX_b8_wordlist,PREFIX_uploads" should do the job.


-- third is the most challenging one. Look at the quoted error. The update script (or better, the database) causes problems when some entries with wrong values are present. In my case, this showed up at the "edited" row of table PREFIX_entries and the "birthday" row of table PREFIX_userdata. In both cases, I had some entries with all zeroed, see above. For some reason, SQL has a problem, depending on the particular settings of the database.

It looks like that these date variables have some default value, if not set. In former times (parsimony?) it was probably "all zero". Now, it seems to be NULL, which is a SQL key word. My SQL database seems to accept either "NULL" for 'not set' or, if set, a certain, realistic value, but not 'all zero'.

Unfortunately, my database let me not change the "all zero" values to an other value. It even refused to accept "all zero" in a query - at least with default settings.

Google pointed me to some database settings, i.e. the "sql_mode". It can be queried like that:

mysql> SHOW VARIABLES LIKE 'sql_mode';
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                                     |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| sql_mode      | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Here, "NO_ZERO_IN_DATE,NO_ZERO_DATE" is causing the issue.

However, I read warnings when changing the settings in a database which is in use. So I created a new database, changed the settings


 SET SESSION sql_mode = 'ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';

(thats just the old settings without "NO_ZERO_IN_DATE,NO_ZERO_DATE"), imported the SQL dump, and changed the "all zero" entries to NULL:


UPDATE PREFIX_entries SET edited = NULL WHERE edited = '0000-00-00 00:00:00';

UPDATE PREFIX_userdata SET birthday = NULL WHERE birthday = '0000-00-00';

Now, a check should not find a "all zero" value any more:


select edited from PREFIX_entries where edited = '0000-00-00 00:00:00';
select birthday from PREFIX_userdata where birthday = '0000-00-00'; 

I made a SQL dump from this change and used it in my testing (and also in my productive) database. It worked without errors.

As indicated, this depends on your individual database settings, so DO NOT SIMPLY COPY&PASTE my hints without looking at your own situation!

After that, I was able to perform the update from 2.4.20 to 20220803.1. At least, the update script finished without error.

Avatar

Update from 2.4.20 to 20220803.1 failed, two issues so far

by Auge ⌂, (798 days ago) @ Nico Hoffmann

Hello Nico,

thank you very much for your investigation and its documentation. 🙏 👍

This gives a clearer picture of the situation. In the long run it is necessary to alter many of the column definitions. Define empty values as NULL instead as 0, 0000-00-00 [00:00:00] or as empty strings would bring us nearer to the safe standards of SQL. I would expect servers to forbid constructions like empty strings or a zero date values but not to forbid NULL values.

Tschö, Auge

--
Trenne niemals Müll, denn er hat nur eine Silbe!

Avatar

For documentation about date default values 0000-00-00

by Auge ⌂, (176 days ago) @ Nico Hoffmann

Hello

While working on an upgrade to a new version, which comes with a completely overhauled update script, I came across this thread again. When redoing the upgrade script I solved every issue that was mentioned in this and also in other threads with the exception of date, datetime and timestamp columns, which once contained 0000-00-00 respectively 0000-00-00-00:00:00 as their default values. I digged into the installation scripts of ancient versions and found definitions like column_name datetime NOT NULL DEFAULT '0000-00-00-00:00:00' in the install.sql of version 2.3.4 as the latest occurence.

With version 2.3.5 the installation script defines all date, datetime or timestamt fields with column_name datetime NULL DEFAULT NULL or column_name datetime NULL DEFAULT CURRENT_TIMESTAMP with one exception where we know, that a date will be given in the queries. That's good for forums, that was installed with version 2.3.5 or later.

In contrast we didn't remove the zero date values which are present in the database with the update script in any version step. That causes very old installations, initially installed with a version before 2.3.5, to possibly show the behaviour, that was reported by Nico because the old definitions and the now (with newer and stricter configured database server versions) broken values remained.

I identified the affected columns and will add queries to redefine these columns as well as set the zero values to NULL.

Tschö, Auge

--
Trenne niemals Müll, denn er hat nur eine Silbe!

RSS Feed of thread