I joined the HHVM team right before Christmas for a hackamonth (or, in my case, a hack-a-two-months).
Getting My Feet Wet
To prepare for what was to be my big project, I rewrote the
ini parser to better match Zend. This was released in HHVM 2.4.0. Interestingly, the parser was going to be shipped with HHVM 2.3.0; but when we tested the RC (release candidate), we noticed that we had a bunch of
ini files at Facebook that the parser failed upon. So, I had to fix those files before we could release the new one.
After warming up with the parser, I was ready to start my big project: implement
MySQLi. This has been a long requested feature for HHVM. And, this extension is required to help meet our compatibility goals.
MySQLi is not a trivial extension. As a PHP developer, I am so glad that SaraG has created HNI (HHVM Native Interface), which makes it possible to write extensions in PHP directly instead of writing them in C++. Also, SaraG created a tool called docskel.php, which uses the raw XML that forms the php.net documentation to stub out the appropriate HNI-based signatures for extensions. While the documentation can be wrong sometimes when it comes to return types and parameter types, the stub generation saves a ton of time. After the stub generation, essentially the only work left is to implement all the required functions.
For many of the MySQLi methods, it was simply a matter of essentially passing through calls to the older MySQL equivalent implementations. For example,
mysqli::real_escape_string() just calls the implementation for
MySQLi has both a procedural and an object-oriented interface. However, because we can write part of the extension in PHP, I could implement almost all of the procedural interface in PHP by just having it call the object-oriented interface. For example, the procedurual function
mysqli_real_escape_string() just calls the object-oriented function
To test the correctness of my implementation, I used the Zend test suite. There are about 300 tests in this test suite. However, these tests are not meant to be run in parallel. Many tests uses the same database table which, when running in parallel, will randomly make the test fail because some other test touched the same table at the same time. Running the tests in serial wasn’t a great option because that would make development of
MySQLi a lot slower. So, instead, I updated our Zend import script to fix the tests to use different tables, thus avoiding the conflicts when running in parallel.
Another testing issue came with what we call
ZendParamMode is our flag to indicate the calling convention that Zend uses. In short, when using
ZendParamMode, if a function is called with a parameter of the wrong type, a warning is logged and the function returns
null. The problem is that functions in
HNI that are implemented in PHP do not currently support
ZendParamMode. So even if the function was doing everything correct, except how it handled
ZendParamMode, the test would fail. This made it very hard to see what was implemented correctly and what was incorrect or missing. But, since making the function actually have correct behavior is more important than supporting
ZendParamMode, I just changed the import script to remove most of the cases where
ZendParamMode was tested.
After fixing the above issues, I was able to perform the normal “where does the test fail and fix it” methodology. This often included reading Zend’s
MySQLi source code to determine what the right behavior should be.
The most difficult part of the
MySQLi implementation was
mysqli_stmt::bind_result(). These methods take a variable number of arguments that are passed by reference. And, at the time,
HNI did not support either. First SaraG added support for variable number of arguments, and then I added support for variables as references.
mysqli_multi_query() (and its friends
mysqli_next_result()) were initially going to be a bit tricky since they were not part of Zend. However, just the opposite occurred. The reason for that is that HHVM already had implementations for
mysql_next_result(), so all I had to do was essentially do a pass through call to those functions.
We currently pass 182 of the Zend
MySQLi tests. We fail 114. However, many of those 114 tests are failures are what I call “technicality” failures. For example, when we
double, we print more digits which makes our output different from Zend.
For real-world compatibility, we pass 100% of doctrine/dbal unit tests and we pass 100% of the CodeIgniter unit tests when running on MySQL.
The biggest missing pieces when it comes to passing all Zend tests are:
Bugs that comes from the interaction between
php.ini. Reading default values from
php.iniis currently missing (but is being developed).
Sometimes we output different error messages which confuses the test runner when it compares output.
Zend has two different
MySQLdrivers. One is the normal
MySQLC API and one is their own
MySQLNative Driver. Because HHVM uses the
MySQLC API we, don’t support the functions that require the
MySQLNative Driver. We may be able to work around this issue for some functions, but, for now, we don’t support any of the functions that require
What happens when something goes wrong may be different between HHVM and Zend.
ZendParamModefor PHP functions in HNI.
Make more of the 114 failing Zend tests pass.
The plan is to release
MySQLiin HHVM 2.5.0. Please test the implementation as much as possible before the 2.5.0 release (you can use a nightly), so we can continue to fix bugs.
- Chase Christian: Thanks WizKid! I was one of the people waiting on the MySQLi and implemented it as soon as your commits were pushed to the nightly build. It has been working great! I haven't had any issues across multiple sites using MySQLi. Thanks again!
- WizKid: Awesome to hear.
- Marco Pivetta: Awesome! I have just enabled HHVM+MySQLi for Doctrine DBAL</a> on Travis-CI</a> as a reference :-) Hope to see green there soon!
- Fred Emmott: This is currently only in the nightly builds, and master, which aren't supported by Travis; these tests will (hopefully :p ) start passing once 2.5.0 is released and Travis upgrade.
- Daniel: I'm looking forward to this!
- Ulf Wendel: Glad to hear that you found the php.net ext/mysqli/tests to be that strict and picky that they make you scratch your head.... Only picky tests can detect changes made without documenting them.
- ms: Any plans for mysqlnd modules like mysqlnd_ms?
- WizKid: Didn't even know that there existed other modules for mysqlnd. Currently there is no plan from our side but we would be happy to take pull requests.
- CJ: "Zend" is a company, not an implementation of PHP.
- Copyright Police: In the world of Facebook, do you not know the name of the programming languages you use? Or is this some sort of twisted way to rebrand PHP as Zend, and this hvvm abomination PHP?
- Paul Tarjan: Zend is the engine that powered PHP4 and Zend 2 powers PHP5 (but most people just call it zend now since PHP4 is retired). https://en.wikipedia.org/wiki/Zend_Engine
- Paul Tarjan: No twisting intended. What would you rather we call things? We need a name for the language (PHP) and the runtime available at php.net. Because the underlying engine is named zend, that is term we usually use: https://en.wikipedia.org/wiki/Zend_Engine Also, why do you think HHVM is an abomination? Are you just trolling or do you have some qualms with our project and there is something we should change?
- Johannes: For nitpicking: There is a company Zend Technologies Ltd., anengine called Zend Engine in different versions and a PHP runtie using it. The Zend Engine can be used for very few things in its own, but you need the PHP parts, like HTTP request handling or different PHP modules. Continueing to nitpick: mysqli is using APIs provided by the Zend Engine but is part of PHP. This can easily eseenin thelicenses headers. The ones in Zend/ say "Zend Engine, Copyright (c) 1998-2014 Zend Technologies Ltd. (http://www.zend.com)" the ones in other parts say " PHP Version 5 , Copyright (c) 1997-2014 The PHP Group" They also usedifferent licenses (Zend Engine License vs. PHP License) But as said this is nitpicking, inpractical terms the software is called PHP, is licensed under PHP Liense (Zend Engine License allows such relicensing) and is developed by a large comunity where even the company Zend Technologies Ltd. is just one between others. Historially the split was more relevant as there was a clear separation between both things and it was more possible to switch out the engine, nowadays, especially with PHP 5 OO APIs this isn't the case anymore. Also historically Zend (the company) was by far the largest contributor to the engine, that has changed and some independent contributors see it as unfair to be pulled in that way. I myself don't have thosestrictfeelings, but some do ...
- Schien: In Zend PHP there are benefits of using MySQLi over MySQL. Is the MySqli extension for HHVM just for parity I suppose? We changed the default db wrapper in our frameworks to MySQL so that they work with HHVM out of the box. There's no need to switch back to MySqli is there? Thanks.
- Paul Tarjan: We don't mean to offend anyone, we just need names for things. What should we call the binary available on php.net? "The reference PHP implementation" is a bit long, "PHP" is confusing to the language, and "php-src" is a bit awkward as a noun.
- Christopher Svanefalk: His name is "Copyright Police", seems like a pretty clearcut troll. Pity they find their way even to great project sites like this one.
- Marco Pivetta: I already enforced DBAL to build on nightly ones - loads of tests are still failing though ;-)