Wow HHVM is fast…too bad it doesn’t run my code

Posted on September 12, 2013 by

HHVM is a highly performant PHP runtime. In fact, it is nearly 40% faster than HPHPc, and only getting faster. For example, the HHVM team just rewrote the JIT to use an SSA intermediate representation (IR). HHIR (HipHop Intermediate Representation) is a strongly typed, SSA-form intermediate representation, positioned between HHBC (HipHop Bytecode) and machine code. It allows HHVM to perform optimizations that were very difficult to perform with the old JIT, using the context of the current runtime environment (e.g., reference counting elision).

HHVM Performance with addition of IR

HHVM Performance with addition of IR

Facebook continues to use fewer resources because of the continual performance increases provided by HHVM. However, there is a problem…

HHVM fatals on Laravel unit tests.

HHVM fatals on Laravel unit tests.

Performance is critical, but it isn’t everything. In order to gain broader adoption for HHVM, being able to run popular frameworks is a must; in other words, we can have the highest performing PHP runtime, but if doesn’t run real-world code without a lot of pain, then it won’t be used widely. Understanding this, we are putting serious resources around parity with the PHP runtime.

The table below shows the team’s current progress towards parity with 20+ of the most popular frameworks used with PHP today. These frameworks come with unit tests that are run via PHPUnit. The unit tests provide a good indication of how well a framework will ultimately run when used in production. The unit tests were run on CentOS 6 from the latest branch of the framework from GitHub, on the latest development build of HHVM.

[table “1” not found /]

It is clear that HHVM is lacking in full support for these frameworks. The first step is to implement the functionality that is causing the HHVM to fatal with some of these unit tests. From the results, in addition to bugs, the HHVM team now knows that it is lacking in support for:

  • Intl (internationalization), MySQLi, ZipArchive, RegexIterator
  • Unimplemented functions such as str_getcsv, stream_context_get_options
  • Richer DOM, Phar and FileInfo support
  • Support for php.ini and associated options
  • Better autoloading support

After ensuring that all unit tests can be run to completion (i.e. no more fatals), the team will continue to implement the functionality that is causing the tests to fail. In parallel, the team will be working to implement any functionality required to allow these frameworks to run on HHVM in the real world (e.g., ensuring the base cases and tutorials can run). Blog posts will be coming soon on how to run the unit tests for each framework on HHVM and/or getting the actual frameworks to run on HHVM (hopefully with less and less workarounds as functionality is implemented).

The HHVM team’s goal is to reach at least parity with the PHP runtime on these (and other) frameworks (the PHP runtime passes the unit tests anywhere from 95-100%)10. Plans are in the works to have a near-live dashboard showing the progress on reaching this goal. If you do not see an important framework in our list that should be added to our testing, please comment below and we will add it.

Moving forward, the HHVM team is going to increase the amount of information around the status of HHVM. Parity status is just one piece of this puzzle. Here is an idea of the plans for the HHVM team in the next 6-12 months:

  • Top PHP Framework parity
  • Updated HHVM documentation
  • First class support for open source contributions
  • Easy install of HHVM on Linux
  • PHP language improvements (e.g., type hints for scalars)

While our small team is aggressively trying to improve HHVM, we know that we cannot address all compatibility concerns as quickly as we would like. Thus, we sincerely welcome contributions, particularly any that can help HHVM reach parity with your desired workload(s). The team will be happy to review and assist with pull requests. Particular thanks to Daniel Sloof, Markus Staab, Vadim Borodavko, Kristaps Kaupe, huzhiguang, and the many others in the community currently pushing commits to our source base.

If you do not have a specific workload or application in mind, but still would like to be involved in HHVM, please look at the issues tagged zend incompatibility for known conflicts with the PHP runtime. Tests can be added easily (see the README) and Travis-CI is integrated, which provides quick and reliable test results on commits. Visit us on IRC channel #hhvm on freenode if you need further guidance.

We know there is work to do, but the HHVM team’s #1 priority right now is being able to run all the (sane) PHP code in the wild.

Footnotes

1. Tests that cause HHVM to fatal were commented out and replaced with a return of $this->fail("HHVM Fatals"); which causes the test to fail, but allow the test suite to continue running.

2. The PHPUnit self-check tests were not run with PHPUnit directly; rather it comes with a self install test that was run, from the source root, via a command similar to ‘hhvm phpunit.php --verbose’. Also, as of now, the tests will only complete when provided with a --debug or --verbose flag; otherwise there is a segmentation fault. That said, PHPUnit does run well enough to be used to execute the unit tests for all the other frameworks.

3. CakePHP has a class named String. HHVM does not allow, by default, for classes to have names of primitives. HHVM has other bugs around preg_replace_callback() and autoloading of classes (https://github.com/facebook/hiphop-php/pull/959).

4. The Joomla unit tests do not currently run because of autoloading issues related to: https://github.com/facebook/hiphop-php/pull/959

5. All 2010 phpBB unit tests fail, and all fail for one reason. There is a method called setValue() that takes two parameters, but throughout the testing only one parameter is provided (and no default value for the second parameter is given). HHVM does not allow this.

6. PHPMyAdmin requires mysqli immediately, and HHVM does not support this.

7. zf2 requires the Intl (internationalization) extension, and HHVM does not support this.

8. Doctrine2 has certain classes that implement interfaces, but these classes do not implement these interfaces fully. HHVM requires classes that implement interfaces to be compatible with the interfaces they implement.

9. A pull request was made for Paris and Idiorm to change their tests to better support HHVM, but to also be more correct: https://github.com/j4mie/idiorm/pull/143

10. The PHP runtime tests were run with a source-code compiled version of PHP 5.5.4 on CentOS 6, with the following configuration:

./configure --prefix=/home/joelm/php55 --with-mysql --with-pdo-mysql --with-curl --with-openssl --with-mcrypt --enable-mbstring --enable-pcntl --with-mysqli --enable-intl --with-pdo-pgsql --with-pgsql --with-libxml-dir --enable-opcache --enable-calendar --enable-sockets --enable-zip --enable-soap --with-zlib --with-gd --with-bz2

11. Magento fatals immediately with the following error message: “HipHop Fatal error: Class Mock_Mage_Core_Controller_Varien_ActionAbstract_839dc612 contains abstract method (dispatch) and must therefore be declared abstract or implement the remaining methods”. We believe it is a problem with the HHVM ReflectionMethod implementation.

Posted in FYI, Parity | Leave a reply

66 Responses to “Wow HHVM is fast…too bad it doesn’t run my code”

  1. Pasquale Vazzana says:

    Keep up the good work!

  2. Yermo says:

    I’ve found the members of your team in IRC extremely helpful and open to bug reports and questions. I confess I was surprised.

    I was convinced there was no way hhvm could run my codebase, a framework/platform not that unlike Drupal, because of how large, involved and most depressingly old it is. To my shock and amazement it runs almost 100%, and I’m confident with a little more effort it will be 100%. And it’s FAST.

    What’s simply nuts is the assertion that in some cases the JIT compiler does a better job (i.e. is faster) than a static C++ compiler can, something that I still have a hard time wrapping my brain around.

  3. Bnaya says:

    I hope namespaces is also on the loop. Thank you for that great project!!

  4. aleksandar says:

    I am sure that including Magento in the “top frameworks” list will be appriciated by it’s large community.

    In case u need help with unit tests let us know :-)

  5. Lelala says:

    The performance issue isn’t quite that important, usually can throw more (cheap today)hardware at the problem.
    Sure, in a datacenter there may be an issue, but in real world, there is not…

    • Rod says:

      Why throw money at a problem instead of using this if you can, for free? I at least hope it’s your own money and not someone else’s.

      • Furicane says:

        Rod, HHVM doesn’t work in real world (yet). There are numerous extensions PHP has that are in use which HHVM doesn’t support and might not support at all. That means you can’t use it. If you have a busy website, that assumes it makes money. Investing money in infrastructure to get several times more back is called smart investing. I just hope no one hired you so far, you seem way too green for real world work.

      • bob says:

        Using this is far from free. You don’t have to pay for a product, but you do have to pay for development time taken to implement this. Then there’s the opportunity cost of taking a few hours (or more) to implement HHVM rather than taking less than an hour to throw more hardware at the problem and using the saved time to do something else.

  6. Awesome stuff. FYI, the sorting on the table doesn’t work – seems to be treating the numbers as strings.

  7. Ilmari Oranen says:

    I’m very excited to see when Drupal unit test pass reaches 100%.

    Supporting the gazillion modules is a different thing, but could be possible with the support of the Drupal community :-)

  8. yosef says:

    What would be very neat of you to add is to make it similar to php5-fpm as possible(allowing us to just as easily add hhvm to replace the php5-fpm) or to simpley run as a seperate server from php5-fpm and await requests of our NGINX.
    so far HHVM configuration has been more confusing than that of PHP5-FPM and I fail to configure it on my own.
    my setup is simple, if the file extension is php then just send it to a unix socket of php5-fpm.sock, the php5-fpm doesn’t care from what folder it is sent to, however hhvm seems like it cares of the paths and folders which makes it quite difficult to figure out how to configure, good work tho and hope you get it beyond your expectations.

  9. Lukas says:

    “Doctrine2 has certain classes that implement interfaces, but these classes do not implement these interfaces fully. HHVM requires classes that implement interfaces to be compatible with the interfaces they implement.”

    could you explain a bit more what you mean by that? see also https://twitter.com/jmikola/status/378374068648419328

  10. It would be nice if you included the methodology for running tests here. I am part of the CakePHP core team, and we’d love to have full compatibility with HipHop, though we can’t do that without knowing how you did setup/ran test :)

    Feel free to ping me if you have any questions.

  11. Ahemd Enan says:

    Great Stuff !, i think wordpress is the most important cms/framework for php and it’s have very large community from different language , please lets us know more about unit tests even we can make especial one for hhvm

  12. Slavin says:

    I honestly fail to see the whole idea of keep trying to make PHP fast. It’s not used in enterprise projects, nor was it intended to be. I have nothing but respect for building (or almost) a blazing fast VM. PHP got so popular because of it’s ease of installation. Take that away and there’s nothing left. Having said that, I do hope that this project actually goes somewhere.

  13. ENTERPRISE OR GTFO says:

    Enterprise projects? You sound like Ballmer.

    You can remove these:
    - WordPress
    - Joomla
    - PHPBB
    - Drupal (< 8)
    - Magento

    Nobody that would care about scaling an application would use such bloated, insecure and poorly architected applications.

  14. Jeff says:

    Looking forward to Magento support!

  15. Alex says:

    +1 To Enterprise or GTFO.

    And forget supporting frameworks – what about MongoDB, Postgres support ?

  16. Daniel Lowrey says:

    Unimplemented functions such as str_getcsv, stream_context_options

    Uh, there’s no such thing as a stream_context_options function. I assume this refers to an inability to use any stream context functionality? If so, this is a major shortcoming for support in my own work.

    Can you please clarify this point?

  17. […] La máquina virtual de Facebook para PHP va aumentando su rendimiento a muy buen ritmo, aunque todavía no tenga soporte total del lenguaje. […]

  18. Yonni says:

    What about Zend Framework 1 & Zend Framework 2? Any relevant test runs with those?

    • Joel Marcey says:

      We have zf2 in the table. It won’t run because of the lack of internationalization support by HHVM. However, we are working on that aspect and working with people who may modify the test suite to help us exit more gracefully than a fatal with the lack of that extension.

  19. I’ve been playing around with HHVM today and it really looks promising!

    The only “hard stop” I had so far is PHPUnit containing un-recognized syntax such as:

    HipHop Fatal error: syntax error, unexpected '(', expecting T_LNUMBER or ';' in /vagrant/ProxyManager/vendor/phpunit/phpunit/PHPUnit/Util/XML.php on line 825

    Had other problems such as the constant PHP_VERSION_ID not being undefined, or missing classes such as RecursiveArrayIterator or missing implementations for the reflection API (mainly seems to crash when you extend internal classes).

    That’s unfortunate, but I can see HipHop VM is now really something easy to install and to use! That surely helps in getting the community to report issues!

    As for the installers: please start signing your .deb packages :-)

  20. Lucas Costa says:

    Thanks! Hope to see you guys around here often!

  21. raciloni says:

    Can you add Fat-Free framework to your table, It’s a realy promising micro-framework, and I’d like to see it supported by HHVM.

  22. RouteXL says:

    I can’t get my Slim project to run on HHVM. Were there any specific configurations made for this framework, or was it ran plain vanilla? I guess there are some rewriterules required to redirect custom endpoints…

  23. John says:

    I agree with Alex. The only reason I can’t use HHVM is because it lacks MongoDB support. There was a promise from Sarah (a year ago or so) that you guys will try to make php extensions easily compilable to HHVM, but still, nothing happened.

  24. If you could only get New Relic Integrated into this I would start transitioning clients tomorrow! I would be scared to lose all the diagnostic information that NR provides my Enterprise Clients

  25. […] HHVM быстр… жаль мой код не запускается — Не смотря на заманчивую производительность, HHVM поддерживает далеко не все возможности PHP. В посте приведена таблица со списком фреймворков и информацией об их работоспособности в HHVM на основе данных о выполнимости юнит-тестов. Приоритетная цель для команды HHVM — обеспечить 100% работу этих фреймворков на HHVM. Помощь контрибьюторов приветствуется. […]

  26. […] couple of weeks ago, a blog post came across /r/php titled Wow HHVM is fast…too bad it doesn’t run my code. The post is pretty interesting, it takes a look at what the test pass % is for a variety of PHP […]

  27. ivan panfilov says:

    I love this project. But really don’t waste time for frameworks and other bullshit stuff. Coz 100% this is not help for improviing performace. Support only standart PHP for people who really needed in perfomance and who writing application
    withaout 100 tons of boilerplate crap.

    • Ivan Mosiev says:

      Please don’t write silly comments. HHVM should support all features of modern PHP, does not matter is it required for frameworks or not. For example we have ArrayAccess interface wich is part of standard PHP library. Unfortunately HHVM does not work correctly with this code:

      $array = new ArrayObject();
      $array['foo'] = 'bar';
      array_key_exists('foo', $array);

      This bug break some tests in PHPUnit and other frameworks and if would like to use ArrayAccess in your code will also break your code too. I think frameworks is used there first of all because of huge amount of unit tests that can be used to verify that HHVM supports all existing PHP features. However, if you mean that you don’t need OOP in PHP at all you are free to use kPHP anaunced not so long time ago, as I know it supports only old school procedural approach.

  28. RouteXL says:

    Referring to my previous question regarding the Slim framework, I got it to work using this config:


    Server {
    SourceRoot = /var/www
    DefaultDocument = index.php
    }

    VirtualHost {
    www {
    Pattern = .*
    ServerVariables {
    PHP_SELF = /index.php
    SCRIPT_NAME = /index.php
    }
    RewriteRules {
    index {
    pattern = ^(.*)$
    to = index.php/$1
    qsa = true
    }
    }
    }
    }

  29. Jakub says:

    I would love to see passing tests for media wiki.

    Thanks for a great project.

  30. […] The differences were smaller and less consistent than I expected. Also I was disappointed that HipHopVM did not run my code. […]

  31. Fabien says:

    Just for your information, I’ve just finished making Twig compatible with HHVM; at least, all tests pass now.

  32. Ryan Kopf says:

    SilverStripe would be another important Framework to test on. SilverStripe powers quite a few government, political, insurance company, and other large websites.

  33. Remo says:

    It seems like concrete5 has a similar problem like CakePHP. There’s a class called “Collection” which causes HHVM to fail with:


    HipHop Fatal error: Class Concrete5_Model_Page may not inherit from interface (Collection) in /home/remo/concrete5/web/concrete/core/models/page.php on line 12

    • Josh says:

      Just tested this on 2.3.0 w/ concrete5 5.6.1.3 – same issue. I’m pretty sure you’re right, it’s colliding due to Collection being a reserved name, as evidenced on line 930 here:
      https://github.com/facebook/hhvm/blob/master/hphp/runtime/vm/class.cpp#L930

      So Collection is already a defined class name and it won’t let you define another (indeed that codepath has instantiated Collection objects of its own already.) If I had to choose, I’d say that the “Collection” is the most fundamental part of Concrete5, so it’s not something easily refactored. It’s old fashioned, but maybe the HHVM guys should consider a prefix to their own naming conventions. hhvmCollection is much less likely to be defined in a PHP script than Collection would be, after all.

      • Paul Tarjan says:

        Collection should already be namespaced. Maybe it didn’t make it into 2.3.0. Can you try it again with trunk?

        • Josh says:

          I found the issue https://github.com/facebook/hhvm/issues/1003 , and it looks like it just missed the cutoff on 2.3.1. Trunk (and everything I see that’s going in to 2.3.2) has this fix included. I’m visiting family for the holidays for the next week, but will definitely test once I’m back to work. It does look like it will all work though – Concrete5′s parity remains to be seen, but this is a huge amount of progress.

  34. Bert-Jan says:

    I would love to see Propel added to the table.. it’s a very popular ORM (like Doctrine) used (among others) in symfony & symfony2, but also stand-alone in other projects.

  35. Greg says:

    Would love to see Guzzle and the AWS SDK supported. Great work so far, very excited to see what the future holds for this project!

  36. grim says:

    Symconf 2 Go~!

  37. […] was a 17% CPU reduction for running facebook.com compared to HHVM 2.1.0. It also supports many more frameworks out of the box like Symfony, CodeIgniter, Laravel, and many […]

  38. Corey Ballou says:

    In future updates, it’d be great if you included the framework version numbers just for reference. Major release versions of frameworks often include substantial changes. For instances, I can see you’re using Laravel 4 for this which was still very much in development at the time so I’m wondering how Laravel 3 faired.

  39. sachin says:

    Can I use it with Core PHP(Procedural style) in any way.I am not using any specific Framework for my application but have a custom built mini framework to perform basic operations.

  40. Fernando says:

    What about Kohana Framework tests? http://kohanaframework.org

    WIll be great to see it!

  41. Philipp Gampe says:

    Any time plan for mysqli support? Mysql is deprecated as of PHP 5.5 and many applications move mysqli.
    http://www.php.net/manual/en/intro.mysql.php

    Also consider adding TYPO3 CMS to the list of tested frameworks – it is the fifth most widely used CMS.
    Unit test examples can be found on travis.

    • Paul Tarjan says:

      mysqli support will land in the next month or two

      • besei says:

        I got to be able to run, “HHVM” program somehow.
        There was an error of, “mysqli” startup screen “phpmyadmin”. I’ve found the cause of the error.
        Thank you.
        Plan and would like to run for example “CakePHP”, “CodeIgniter”, “FuelPHP”.

        These comments, is the google translation.

  42. Philipp Gampe says:

    I just checked … old stable TYPO3 CMS Version 4.5 LTS runs with hhvm <3 in frontend context.

    This is really fast.

  43. […] September 2013 the PHP HipHop VM dev team wrote a very interesting article [1] about failing Unit Tests of most PHP frameworks when running inside the HipHop VM (HHVM). The […]

  44. Loyd says:

    My programmer is trying to convince me to move to .net from PHP.
    I have always disliked the idea because of the expenses.

    But he’s tryiong none the less. I’ve been using Movable-type on a number of websites for about a
    year and am anxious about switching to another platform.

    I have heard excellent things about blogengine.net.
    Is there a way I can transfer all my wordpress posts into it?
    Any kind of help would be really appreciated!

  45. […] Wow HHVM is fast…too bad it doesn’t run my code↩ […]

  46. […] 20+ top PHP frameworks have 100% unit tests passing […]

  47. […] engine. This is necessary because Facebook’s existing codebase is largely existing PHP code. There are a few exceptions since HHVM is not 100% compatible with the PHP Zend Engine; there are a few language features (like […]

Leave a Reply