Wow HHVM is fast...too bad it doesn’t run my code

HHVM is a highly performant PHP runtime. In fact, it is nearly 40% faster than HPHPc, and only getting faster. For example, the HHVM team just rewrote the JIT to use an SSA intermediate representation (IR). HHIR (HipHop Intermediate Representation) is a strongly typed, SSA-form intermediate representation, positioned between HHBC (HipHop Bytecode) and machine code. It allows HHVM to perform optimizations that were very difficult to perform with the old JIT, using the context of the current runtime environment (e.g., reference counting elision).

Facebook continues to use fewer resources because of the continual performance increases provided by HHVM. However, there is a problem…

Performance is critical, but it isn’t everything. In order to gain broader adoption for HHVM, being able to run popular frameworks is a must; in other words, we can have the highest performing PHP runtime, but if doesn’t run real-world code without a lot of pain, then it won’t be used widely. Understanding this, we are putting serious resources around parity with the PHP runtime.

The table below shows the team’s current progress towards parity with 20+ of the most popular frameworks used with PHP today. These frameworks come with unit tests that are run via PHPUnit. The unit tests provide a good indication of how well a framework will ultimately run when used in production. The unit tests were run on CentOS 6 from the latest branch of the framework from GitHub, on the latest development build of HHVM.

It is clear that HHVM is lacking in full support for these frameworks. The first step is to implement the functionality that is causing the HHVM to fatal with some of these unit tests. From the results, in addition to bugs, the HHVM team now knows that it is lacking in support for:

• Intl (internationalization), MySQLi, ZipArchive, RegexIterator

• Unimplemented functions such as str_getcsv, stream_context_get_options

• Richer DOM, Phar and FileInfo support

• Support for php.ini and associated options

• Better autoloading support

After ensuring that all unit tests can be run to completion (i.e. no more fatals), the team will continue to implement the functionality that is causing the tests to fail. In parallel, the team will be working to implement any functionality required to allow these frameworks to run on HHVM in the real world (e.g., ensuring the base cases and tutorials can run). Blog posts will be coming soon on how to run the unit tests for each framework on HHVM and/or getting the actual frameworks to run on HHVM (hopefully with less and less workarounds as functionality is implemented).

The HHVM team’s goal is to reach at least parity with the PHP runtime on these (and other) frameworks (the PHP runtime passes the unit tests anywhere from 95-100%) [11]. Plans are in the works to have a near-live dashboard showing the progress on reaching this goal. If you do not see an important framework in our list that should be added to our testing, please comment below and we will add it.

Moving forward, the HHVM team is going to increase the amount of information around the status of HHVM. Parity status is just one piece of this puzzle. Here is an idea of the plans for the HHVM team in the next 6-12 months:

• Top PHP Framework parity

• Updated HHVM documentation

• First class support for open source contributions

• Easy install of HHVM on Linux

• PHP language improvements (e.g., type hints for scalars)

While our small team is aggressively trying to improve HHVM, we know that we cannot address all compatibility concerns as quickly as we would like. Thus, we sincerely welcome contributions, particularly any that can help HHVM reach parity with your desired workload(s). The team will be happy to review and assist with pull requests. Particular thanks to Daniel Sloof, Markus Staab, Vadim Borodavko, Kristaps Kaupe, huzhiguang, and the many others in the community currently pushing commits to our source base.

If you do not have a specific workload or application in mind, but still would like to be involved in HHVM, please look at the issues tagged zend incompatibility for known conflicts with the PHP runtime. Tests can be added easily (see the README) and Travis-CI is integrated, which provides quick and reliable test results on commits. Visit us on IRC channel #hhvm on freenode if you need further guidance.

We know there is work to do, but the HHVM team’s #1 priority right now is being able to run all the (sane) PHP code in the wild.

Footnotes

1. Tests that cause HHVM to fatal were commented out and replaced with a return of $this->fail("HHVM Fatals"); which causes the test to fail, but allow the test suite to continue running.

2. The PHPUnit self-check tests were not run with PHPUnit directly; rather it comes with a self install test that was run, from the source root, via a command similar to ‘hhvm phpunit.php --verbose’. Also, as of now, the tests will only complete when provided with a --debug or --verbose flag; otherwise there is a segmentation fault. That said, PHPUnit does run well enough to be used to execute the unit tests for all the other frameworks.

3. CakePHP has a class named String. HHVM does not allow, by default, for classes to have names of primitives. HHVM has other bugs around preg_replace_callback() and autoloading of classes (https://github.com/facebook/hiphop-php/pull/959).

4. The Joomla unit tests do not currently run because of autoloading issues related to: https://github.com/facebook/hiphop-php/pull/959

5. All 2010 phpBB unit tests fail, and all fail for one reason. There is a method called setValue() that takes two parameters, but throughout the testing only one parameter is provided (and no default value for the second parameter is given). HHVM does not allow this.

6. PHPMyAdmin requires mysqli immediately, and HHVM does not support this.

7. zf2 requires the Intl (internationalization) extension, and HHVM does not support this.

8. Doctrine2 has certain classes that implement interfaces, but these classes do not implement these interfaces fully. HHVM requires classes that implement interfaces to be compatible with the interfaces they implement.

9. A pull request was made for Paris and Idiorm to change their tests to better support HHVM, but to also be more correct: https://github.com/j4mie/idiorm/pull/143

10. Magento fatals immediately with the following error message: “HipHop Fatal error: Class Mock_Mage_Core_Controller_Varien_ActionAbstract_839dc612 contains abstract method (dispatch) and must therefore be declared abstract or implement the remaining methods”. We believe it is a problem with the HHVM ReflectionMethod implementation.

11. The PHP runtime tests were run with a source-code compiled version of PHP 5.5.4 on CentOS 6, with the following configuration:

./configure --prefix=/home/joelm/php55 --with-mysql --with-pdo-mysql --with-curl --with-openssl --with-mcrypt --enable-mbstring --enable-pcntl --with-mysqli --enable-intl --with-pdo-pgsql --with-pgsql --with-libxml-dir --enable-opcache --enable-calendar --enable-sockets --enable-zip --enable-soap --with-zlib --with-gd --with-bz2

Comments

• Pasquale Vazzana: Keep up the good work!
• Yermo: I've found the members of your team in IRC extremely helpful and open to bug reports and questions. I confess I was surprised. I was convinced there was no way hhvm could run my codebase, a framework/platform not that unlike Drupal, because of how large, involved and most depressingly old it is. To my shock and amazement it runs almost 100%, and I'm confident with a little more effort it will be 100%. And it's FAST. What's simply nuts is the assertion that in some cases the JIT compiler does a better job (i.e. is faster) than a static C++ compiler can, something that I still have a hard time wrapping my brain around.
• Bnaya: I hope namespaces is also on the loop. Thank you for that great project!!
• aleksandar: I am sure that including Magento in the "top frameworks" list will be appriciated by it's large community. In case u need help with unit tests let us know :-)
• Lelala: The performance issue isn't quite that important, usually can throw more (cheap today)hardware at the problem. Sure, in a datacenter there may be an issue, but in real world, there is not...
• Joseph Redfern: Awesome stuff. FYI, the sorting on the table doesn't work - seems to be treating the numbers as strings.
• Ilmari Oranen: I'm very excited to see when Drupal unit test pass reaches 100%. Supporting the gazillion modules is a different thing, but could be possible with the support of the Drupal community :-)
• yosef: What would be very neat of you to add is to make it similar to php5-fpm as possible(allowing us to just as easily add hhvm to replace the php5-fpm) or to simpley run as a seperate server from php5-fpm and await requests of our NGINX. so far HHVM configuration has been more confusing than that of PHP5-FPM and I fail to configure it on my own. my setup is simple, if the file extension is php then just send it to a unix socket of php5-fpm.sock, the php5-fpm doesn't care from what folder it is sent to, however hhvm seems like it cares of the paths and folders which makes it quite difficult to figure out how to configure, good work tho and hope you get it beyond your expectations.
• Lukas: "Doctrine2 has certain classes that implement interfaces, but these classes do not implement these interfaces fully. HHVM requires classes that implement interfaces to be compatible with the interfaces they implement." could you explain a bit more what you mean by that? see also https://twitter.com/jmikola/status/378374068648419328
• Jose Diaz-Gonzalez: It would be nice if you included the methodology for running tests here. I am part of the CakePHP core team, and we'd love to have full compatibility with HipHop, though we can't do that without knowing how you did setup/ran test :) Feel free to ping me if you have any questions.
• Ahemd Enan: Great Stuff !, i think wordpress is the most important cms/framework for php and it's have very large community from different language , please lets us know more about unit tests even we can make especial one for hhvm
• Joel Marcey: aleksandar, thanks for your comment. I have added Magento to the table with a footnote as to why the unit tests won't run. We are pretty sure it is a bug on our end. Once that is fixed, we will see what other failures we may have :)
• Joel Marcey: Jose Diaz-Gonzalez, I am planning to update our GitHub wiki with information on the setup and processes we used to run these tests. I hope to get that documented very soon.
• Sam: How about Guzzle? http://guzzlephp.org/
• Slavin: I honestly fail to see the whole idea of keep trying to make PHP fast. It's not used in enterprise projects, nor was it intended to be. I have nothing but respect for building (or almost) a blazing fast VM. PHP got so popular because of it's ease of installation. Take that away and there's nothing left. Having said that, I do hope that this project actually goes somewhere.
• ENTERPRISE OR GTFO: Enterprise projects? You sound like Ballmer. You can remove these: - Wordpress - Joomla - PHPBB - Drupal (< 8) - Magento Nobody that would care about scaling an application would use such bloated, insecure and poorly architected applications.
• Jeff: Looking forward to Magento support!
• Alex: +1 To Enterprise or GTFO. And forget supporting frameworks - what about MongoDB, Postgres support ?
• Daniel Lowrey:

  Unimplemented functions such as str_getcsv, stream_context_options</blockquote> Uh, there's no such thing as a stream_context_options</code> function. I assume this refers to an inability to use any</i> stream context functionality? If so, this is a major shortcoming for support in my own work. Can you please clarify this point?

• Joel Marcey: Good catch, thanks. It should have been stream_context_get_options</code>. I fixed it.
• Noticias 14-10-2013 - La Web de Programación: […] La máquina virtual de Facebook para PHP va aumentando su rendimiento a muy buen ritmo, aunque todavía no tenga soporte total del lenguaje. […]
• Yonni: What about Zend Framework 1 & Zend Framework 2? Any relevant test runs with those?
• Marco Pivetta: I've been playing around with HHVM today and it really looks promising! The only "hard stop" I had so far is PHPUnit containing un-recognized syntax such as: HipHop Fatal error: syntax error, unexpected '(', expecting T_LNUMBER or ';' in /vagrant/ProxyManager/vendor/phpunit/phpunit/PHPUnit/Util/XML.php on line 825</code> Had other problems such as the constant PHP_VERSION_ID</code> not being undefined, or missing classes such as RecursiveArrayIterator</code> or missing implementations for the reflection API (mainly seems to crash when you extend internal classes). That's unfortunate, but I can see HipHop VM is now really something easy to install and to use! That surely helps in getting the community to report issues! As for the installers: please start signing your .deb packages :-)
• Lucas Costa: Thanks! Hope to see you guys around here often!
• Joel Marcey: We have zf2 in the table. It won't run because of the lack of internationalization support by HHVM. However, we are working on that aspect and working with people who may modify the test suite to help us exit more gracefully than a fatal with the lack of that extension.
• raciloni: Can you add Fat-Free framework</a> to your table, It's a realy promising micro-framework, and I'd like to see it supported by HHVM.
• RouteXL: I can't get my Slim project to run on HHVM. Were there any specific configurations made for this framework, or was it ran plain vanilla? I guess there are some rewriterules required to redirect custom endpoints...
• John: I agree with Alex. The only reason I can't use HHVM is because it lacks MongoDB support. There was a promise from Sarah (a year ago or so) that you guys will try to make php extensions easily compilable to HHVM, but still, nothing happened.
• Brian Cavanagh: If you could only get New Relic Integrated into this I would start transitioning clients tomorrow! I would be scared to lose all the diagnostic information that NR provides my Enterprise Clients
• Rod: Why throw money at a problem instead of using this if you can, for free? I at least hope it's your own money and not someone else's.
• Дайджест интересных новостей и материалов из мира PHP за последние две недели (8—22 сентября 2013) | Juds: […] HHVM быстр… жаль мой код не запускается — Не смотря на заманчивую производительность, HHVM поддерживает далеко не все возможности PHP. В посте приведена таблица со списком фреймворков и информацией об их работоспособности в HHVM на основе данных о выполнимости юнит-тестов. Приоритетная цель для команды HHVM — обеспечить 100% работу этих фреймворков на HHVM. Помощь контрибьюторов приветствуется. […]
• PHP: Installing HipHop PHP on Ubuntu « {5} Setfive – Talking to the World: […] couple of weeks ago, a blog post came across /r/php titled Wow HHVM is fast…too bad it doesn’t run my code. The post is pretty interesting, it takes a look at what the test pass % is for a variety of PHP […]
• ivan panfilov: I love this project. But really don't waste time for frameworks and other bullshit stuff. Coz 100% this is not help for improviing performace. Support only standart PHP for people who really needed in perfomance and who writing application withaout 100 tons of boilerplate crap.
• RouteXL: Referring to my previous question regarding the Slim framework, I got it to work using this config: Server { SourceRoot = /var/www DefaultDocument = index.php } VirtualHost { www { Pattern = .* ServerVariables { PHP_SELF = /index.php SCRIPT_NAME = /index.php } RewriteRules { index { pattern = ^(.*)$ to = index.php/$1 qsa = true } } } } </code>
• Jakub: I would love to see passing tests for media wiki. Thanks for a great project.
• Benchmarking HipHopVM against PHP 5.3 on Ubuntu 12.04 | LeaseWeb Labs: […] The differences were smaller and less consistent than I expected. Also I was disappointed that HipHopVM did not run my code. […]
• Fabien: Just for your information, I've just finished making Twig compatible with HHVM; at least, all tests pass now.
• Ryan Kopf: SilverStripe would be another important Framework to test on. SilverStripe powers quite a few government, political, insurance company, and other large websites.
• Remo: It seems like concrete5 has a similar problem like CakePHP. There's a class called "Collection" which causes HHVM to fail with: HipHop Fatal error: Class Concrete5_Model_Page may not inherit from interface (Collection) in /home/remo/concrete5/web/concrete/core/models/page.php on line 12</code>
• Bert-Jan: I would love to see Propel added to the table.. it's a very popular ORM (like Doctrine) used (among others) in symfony & symfony2, but also stand-alone in other projects.
• Andrey Tserkus: Can you also add Magento 2 to the list? It is a newer version under development, it has more unit tests. GitHub link: https://github.com/magento/magento2 Thanks!
• Greg: Would love to see Guzzle and the AWS SDK supported. Great work so far, very excited to see what the future holds for this project!
• grim: Symconf 2 Go~!
• HHVM 2.2.0 « HipHop Virtual Machine: […] was a 17% CPU reduction for running facebook.com compared to HHVM 2.1.0. It also supports many more frameworks out of the box like Symfony, CodeIgniter, Laravel, and many […]
• Corey Ballou: In future updates, it'd be great if you included the framework version numbers just for reference. Major release versions of frameworks often include substantial changes. For instances, I can see you're using Laravel 4 for this which was still very much in development at the time so I'm wondering how Laravel 3 faired.
• Ivan Mosiev: Please don't write silly comments. HHVM should support all features of modern PHP, does not matter is it required for frameworks or not. For example we have ArrayAccess interface wich is part of standard PHP library. Unfortunately HHVM does not work correctly with this code: $array = new ArrayObject(); $array['foo'] = 'bar'; array_key_exists('foo', $array); </code> This bug break some tests in PHPUnit and other frameworks and if would like to use ArrayAccess in your code will also break your code too. I think frameworks is used there first of all because of huge amount of unit tests that can be used to verify that HHVM supports all existing PHP features. However, if you mean that you don't need OOP in PHP at all you are free to use kPHP anaunced not so long time ago, as I know it supports only old school procedural approach.
• Paul Tarjan: Have you reported the bug on github? And we would love a pull request :)
• Ivan Mosiev: Report no problem, but about pull request I'm unsure. I'm not so good in C++
• sachin: Can I use it with Core PHP(Procedural style) in any way.I am not using any specific Framework for my application but have a custom built mini framework to perform basic operations.
• Paul Tarjan: Yes
• Fernando: What about Kohana Framework tests? http://kohanaframework.org WIll be great to see it!
• Furicane: Rod, HHVM doesn't work in real world (yet). There are numerous extensions PHP has that are in use which HHVM doesn't support and might not support at all. That means you can't use it. If you have a busy website, that assumes it makes money. Investing money in infrastructure to get several times more back is called smart investing. I just hope no one hired you so far, you seem way too green for real world work.
• bob: Using this is far from free. You don't have to pay for a product, but you do have to pay for development time taken to implement this. Then there's the opportunity cost of taking a few hours (or more) to implement HHVM rather than taking less than an hour to throw more hardware at the problem and using the saved time to do something else.
• Josh: Just tested this on 2.3.0 w/ concrete5 5.6.1.3 - same issue. I'm pretty sure you're right, it's colliding due to Collection being a reserved name, as evidenced on line 930 here: https://github.com/facebook/hhvm/blob/master/hphp/runtime/vm/class.cpp#L930 So Collection is already a defined class name and it won't let you define another (indeed that codepath has instantiated Collection objects of its own already.) If I had to choose, I'd say that the "Collection" is the most fundamental part of Concrete5, so it's not something easily refactored. It's old fashioned, but maybe the HHVM guys should consider a prefix to their own naming conventions. hhvmCollection is much less likely to be defined in a PHP script than Collection would be, after all.
• Paul Tarjan: Collection should already be namespaced. Maybe it didn't make it into 2.3.0. Can you try it again with trunk?
• Rafi B.: +1 for Kohana, would be interesting.
• Philipp Gampe: Any time plan for mysqli support? Mysql is deprecated as of PHP 5.5 and many applications move mysqli. http://www.php.net/manual/en/intro.mysql.php Also consider adding TYPO3 CMS to the list of tested frameworks - it is the fifth most widely used CMS. Unit test examples can be found on travis.
• Philipp Gampe: I just checked ... old stable TYPO3 CMS Version 4.5 LTS runs with hhvm <3 in frontend context. This is really fast.
• Paul Tarjan: mysqli support will land in the next month or two
• Josh: I found the issue https://github.com/facebook/hhvm/issues/1003 , and it looks like it just missed the cutoff on 2.3.1. Trunk (and everything I see that's going in to 2.3.2) has this fix included. I'm visiting family for the holidays for the next week, but will definitely test once I'm back to work. It does look like it will all work though - Concrete5's parity remains to be seen, but this is a huge amount of progress.
• HipHop VM reaches 100% green Unit Tests in Laravel, Drupal, Slim, CodeIgniter etc. | Dev Metal: […] September 2013 the PHP HipHop VM dev team wrote a very interesting article [1] about failing Unit Tests of most PHP frameworks when running inside the HipHop VM (HHVM). The […]
• Loyd: My programmer is trying to convince me to move to .net from PHP. I have always disliked the idea because of the expenses. But he's tryiong none the less. I've been using Movable-type on a number of websites for about a year and am anxious about switching to another platform. I have heard excellent things about blogengine.net. Is there a way I can transfer all my wordpress posts into it? Any kind of help would be really appreciated!
• HHVM and WordPress | Keita's Blog: […] Wow HHVM is fast…too bad it doesn’t run my code↩ […]
• HHVM: The Next Six Months « HipHop Virtual Machine: […] 20+ top PHP frameworks have 100% unit tests passing […]
• besei: I got to be able to run, "HHVM" program somehow. There was an error of, "mysqli" startup screen "phpmyadmin". I've found the cause of the error. Thank you. Plan and would like to run for example "CakePHP", "CodeIgniter", "FuelPHP". These comments, is the google translation.
• Facebook's Hack, HHVM, and the Future of PHP - Programming - O'Reilly Media: […] engine. This is necessary because Facebook’s existing codebase is largely existing PHP code. There are a few exceptions since HHVM is not 100% compatible with the PHP Zend Engine; there are a few language features (like […]
• Hubnest » What is HHVM?: […] not for everybody. Although it runs all our codes out of the box, HHVM’s PHP implementation isn’t complete enough to run many popular PHP frameworks. One of the issues is the server requirements – HHVM is […]
• Pavel: Looks very promising but a bit diffucult to deal with some configuration problems. We need to disable php native soapclient because we use nusoap. We failed miserably. Is there a way to disable some soap or some other modules? Traditional way does not work. Is there something fundamental we are missing?
• Major: +1 for Kohana!!!!!
• John Sutton: I used HPHPc and I liked it. Then HHVM came along and I was disappointed since it no longer compiled to C, but that decision wasn't mine. I avoided HHVM as long as I could, but because of Ubuntu upgrades I eventually installed HHVM and FastCGI. When I tried it I discovered HHVM does not support ODBC_CONNECT - a very commonly used function. I have many PHP webservices that pull data using ODBC. I am really disappointed.
• Theodore R. Smith: Hey John, the many database-specific functions from the 1990s are no longer wise to rely on. Indeed, they're starting to be deprecated by Zend PHP itself. Today, in 2014, you really really ought to migrate to PDO (which does support ODBC), the de facto standard for database access.
• k0d3g3ar: How about losing the personal attacks here and actually contributing something that we all can benefit from. Telling someone that they shouldn't be hired or are too green makes you look like someone I would never hire - not a team player.
• k0d3g3ar: It is used in enterprise projects - lots of them. Also PHP lacks in any form of true compilation to hide source code. This is a highly needed and valuable project.
• k0d3g3ar: +1
• HHVM will kompatibel mit den häufigsten PHP Frameworks werden - entwickler.de: […] man in den vergangenen Monaten große Fortschritte bei der Performance gemacht hat, wie aus der aktuellen Meldung hervorgeht, will man nun mit höchster Priorität an der Kompatibilität zu Anwendungen aus der […]