If you’ve looked at my previous entry Getting Wordpress Running on HHVM, you’ve already got a general idea for how to create a config.hdf file and get a web server up and running. Now you want to make sure it’s running as quickly, and efficiently as possible. But first, it’s worth understanding how HHVM runs your PHP scripts.

How HHVM serves up a page

When a request comes in, HHVM considers the hostname of the request against VirtualHost and Sandbox configuration directives to determine the right base directory to serve files from.

Once it’s identified the file, it checks a code cache stored in a SQLite database to see if it has already compiled the file. If so, and the file hasn’t changed, HHVM will execute that cached version very similarly to how APC caches PHP’s compilation process. If not, then HHVM’s analysis engine is invoked to compile the script, perform type inference, optimization, and store the result for next time.

As you might expect, there is a short warm-up period when a new server is started with a fresh, empty cache. Just like any other compilation cache, this only slows the first request of a given page. Unlike APC, however, HHVM’s cache is on disk, and therefore survives restarts.

If you’re curious about the code cache repository, you can find it in the location specified by the setting Repo.Central.Path, which defaults to ~/.hhvm.hhbc (you probably want to change that). Refer to doc/repo for more info on Repo settings.

1
2
3
4
5

Repo {
  Central {
    Path = /var/run/hhvm.hhbc.sq3
  }
}

Just In Time

For production servers, you’ll almost certainly want to enable JIT (Just in Time) compilation to progressively turn the bytecode contained in hhvm.hhbc.sq3 into native code. Depending on exactly what your code base is doing, this will result in performance improvements of anywhere from 50% to 300% (100% seems to be a fairly common average).

To do so, just add the following snippet to your config.hdf file. Yes, it really is a “go faster” button.

1
2
3

Eval {
  Jit = true
}

Bear in mind that JIT compilation is meant for long-running web servers where the extra effort of compiling to native code pays off after a few page views. For single-run uses like command line scripts, the extra work of compiling to native code will actually slow it down compared to just executing bytecode.

Note that as of Oct 18, 2012, HHVM will enable JIT by default for server, daemon, and replay modes, and disable by default for cli, debug, and translate modes.  Ref: 57f063d2 e5824bea

Pre-analyzing

As stated already, all compilation from source to bytecode is cached on disk, so restarting your web server doesn’t lose any of that information and there will be no “warm-up” period on your second and later server starts; But why have that first warm-up period at all?

When you built HipHop (with USE_HHVM=1), the build process actually made two binaries. One, in src/hhvm/hhvm, is the web server and PHP engine you know and love. The other, in src/hphp/hphp (aka hhvm-analyze), is a version of HPHPc meant for HHVM use. It’s this latter binary which will allow you to pre-generate that cache before your web server ever starts up.

If you’re using our pre-built _hiphop-php_ package for Ubuntu 12.04, this HHVM version of hphp has been delivered as** /usr/bin/hhvm-analyze. The name has been changed to avoid confusion with **/usr/bin/hphp which will be the normal HPHPc compiler in non-HHVM mode. For the purpose of this article, I’ll refer to running hhvm-analyze. Those who compiled HipHop themselves should take this to mean src/hphp/hphp built using USE_HHVM=1

To build the cache, you’ll need to identify the files you want included and invoke the analyzer:

1
2
3
4
5

$ cd /var/www
$ find . -name '*.php' > /tmp/files.list
$ hhvm --hphp --target hhbc --input-list /tmp/files.list --output-file hhvm.hhbc.sq3 -k1 -l3

The newly generated bytecode cache will be in <strong>/tmp/hphp_XXXXX/hhvm.hhbc.sq3</strong> where XXXXX is a unique label produced in the output.

In addition to avoiding the initial warm-up by doing that ahead of time, hhvm-analyze will actually work a little harder than hhvm would at runtime. It’ll find a few more optimizations, be a little more sure about type inference, and produce better bytecode.

Repo.Authoritative

Even with the cache completely primed by running the analysis step above, HHVM is still going to default to checking the subject file each time it’s requested to make sure it hasn’t changed. That means disk I/O, and that means time. For a production server where source files are confidently not-changing, that’s just a waste. APC users will be familiar with this concept from the “apc.stat=0” option.

To ignore the state of the source files during runtime on your production servers, just tell HHVM that the code cache (or Repo in HipHop parlance) is the authoritative source of all scripts on the system. With this setting enabled, if the file is found in the cache, it’s used without checking for updates. If it’s not found in the cache, the server will offer up a 404 without even trying to grab the file.

1
2
3

Repo {
  Authoritative = true
}

Note that the bytecode cache is also tagged with the build ID of the hhvm which generated it. If a new hhvm binary is used, it will invalidate any previous cache and start fresh. So when running in Repo/Authoritative mode, be sure to regenerate your cache any time you replace your hhvm binary.

Comments


  • Michael Stillwell: Thanks--this is very useful! And it seems as though HipHop is getting closer and closer to becoming a faster version of 5.4's inbuilt webserver... Do you have any information on using HipHop (hhvm or hhvm-analyze) for static analysis? What target should be used to generate a meaningful CodeError.js?
  • sgolemon: Hrmm, I'm not sure what you mean by "5.4's inbuilt webserver" since the official PHP (php.net) doesn't have an inbuilt web server. But if you want to compare PHP+Apache-prefork (most common configuration of PHP), then HHVM is already several times faster. I have some graphs I've been preparing for a blog entry covering the methodology used and the results observed. As for static analysis, an article on that is coming as well. Watch this space!
  • Michael Stillwell: Regarding the webserver, I mean the -S option: "-S : Run with built-in web server." It's surprising how often I find myself wanting this, for a quick hack or testing. Looking forward to the static analysis article!
  • Sandeepone: Heads up for the info. Very informative and saves time without digging into hphp code.. I have a question, hiphop is not using all the php 5.3 features yet. Is there any timeframe? or some features wont be supported like arrayobject class and namespaces. Thank you!
  • sgolemon: Our intention is to reach (nearly) 100% parity with php.net. I say nearly because there will almost certainly be one or two features where we disagree, but we're aiming towards 100%. As far as timeline goes... individual features will be worked on as time permits. If there's a particular feature you need, speak up and we can try and prioritize it. :)
  • sgolemon: Color me surprised, I didn't see this go in last year. I'll have to add this to my performance graphs. git show 5b921a87
  • Maxi: Facebook HipHop is really interesting. I want to see the performance graphics!
  • Sandeepone: Thanks. Nothing in specific, mainly namespaces, rest of the things can be managed.
  • sgolemon: Yeah, namespaces is a sore point for me as well. At the moment there's not a clean way to add them without causing slow-downs. However once we're able to remove HPHPc and focus exclusively on HHVM, we have some plans to add namespaces in a way that will be performance neutral if you don't use them, and only minimally costly if you do.
  • Jefferson: Does JIT works correctly with eval? I'm testing a CMS I developed that uses eval for loading code of page sections and hhvm seems to run pretty well, but after opening some pages hhvm seg faults (with JIT=false it doesn't segfaults), is there some verbosity option or debugging facility? Also does including php files dynamically works correctly? lets said, I have a clean url rule that says index.php should load /section/content from ./section/load.php, the same would apply for a bunch of other sections. Each time index.php with a new section the file is re-read? In a system that is modular (plugins) and functionality could be enabled on demand? how is that processed? Lets say I activate a module on drupal does it is proccesed by JIT correctly? does include() function checks if a file was already converted to native code? I have noticed hhvm to be slower than original php even with JIT turned on, but again my code uses eval a lot, not sure if thats the issue.
  • Jefferson: After doing some more testing I have to correct what I said before, hhvm is about 20% more faster than php + xcache with JIT set to false, with JIT my code performance increased 9% more, but it maybe due to the extragavant use of eval on it. And as I said before some code causes hhvm segfaults with JIT = true. This project has become a real alternative to original php runtime. It rocks! Now I would like to write some extensions for developing GUI applications with it :)
  • sgolemon: JIT+eval? Yes, all code goes through the same bytecode compiler and from there to the same JIT compiler. The only difference between eval() and include() is where in the bytecode cache the intermediate representation sits, but that's a minor implementation detail. Dynamic includes: Absolutely. Wordpress uses these for its plugins and this site manages that on HHVM just fine. When a given file is included the logic is (roughly): 1. Has the file changed since the last time it was included, if so skip to step 5.* 2. Do we have native code built already? If so run it. [DONE] 3. Do we already have bytecode that's been hit a few times**? Send it to the JIT, run the resulting native code and save it for later. [DONE] 4. Do we have bytecode which isn't hit frequently? Run it via the VM as bytecode. 5. Compile the source to bytecode and run it via the VM.* * - Unless we're in RepoAuthoritative mode, in which case we don't consult the source code. ** - The decision on when to JIT is more complicated than that, having to do with system load, frequency of hits, and other factors. eval() as a factor? It depends on how dynamic your eval is, though as Rasmus is oft quoted: "If eval is the answer, you're asking the wrong question." If the eval'd string is different each time, then it has to be compiled each time, but regular PHP suffers the same hit here, so it shouldn't make a huge difference. As to the definition of "faster", it's important to talk about the same scale. Blocking operations like SQL queries or file/network I/O are going to be an equal hit for both platforms. If you look at active CPU time (and memory usage) you'll probably see a more significant gain between regular PHP and HHVM. This is significant for more than just academic reasons since if you're spending half as much actual CPU time per request, that means that you can handle twice as many clients per front-end machine***. An individual request may only be 10 or 20% faster, but the whole is delivering a much higher RPS throughput. *** Assuming of course that your DB layer can keep up. Scaling that part of your architecture is an entirely separate conversation. Also bear in mind that PHP doesn't play well with apache2-worker (threaded mode) so every parallelization of request handling is another process. HHVM is a single process multi-thread webserver and has a much lower per-child memory footprint compared to Apache+PHP. As to crashing when JIT is enabled: That (obviously) shouldn't happen. If you can come up with a repro case we might be able to identify the cause and get a fix out for it.
  • Jefferson: Thanks a lot for the valuable explanation. So in other words it can be said, for example, that the md5 of a string used on an eval() is saved in order to associate the bytecode generated for it so future calls to eval with a string of the same md5 invoke that bytecode, thats great to know. Doing some testing I noticed that hhvm with my custom cms only uses about 2 megs of ram and 1% of cpu (AMD Phenom(tm) II X4 95) per request in contrast with original php that can use about 18-20 megs of ram and 15% of cpu. So the gain is huge! As for the JIT crashes I was looking at the documentation files found in the hiphop source tree and I will try to enable some debugging settings and see what is causing them. Thanks
  • sgolemon: Yep. If you look in your hhbc file you'll find the eval'd bytecode units and they'll be indexed on a hash just as you guessed. gdb might help you nail down the segfault to a call-tree at least...
  • Matt: Morning, I would like to confirm what you do next after analysing the /var/www directory. I followed your steps as outlined above and got one error message asking me to add a path to home (export HPHP_HOME='pwd'). However after this there is a hhvm.hhbc file created in the new temp dir but not the .sq3 file. The message tells me there are some files missing but these files are not on my ubuntu 12.04 install anywhere (i installed hiphop via the package). Can you let me know how this can be fixed please. Also, once the .sql3 file is produced im guessing that i just replace the one made from the config file (/etc/hhvm.hdf). Thanks for your help. Once i get this working i will report the speed improvemenets. So far i notice a big drop in cpu use and ram used but the speeds to load the page is pretty similar so far. Cheers hhvm-analyze --target hhbc --input-list /tmp/files.list --output-file hhvm.hhbc.sq3 -k1 -l3 running hphp... creating temporary directory /tmp/hphp_3wwUN3 ... Unable to open file pwd/src/system/classes/stdclass.php Unable to open file pwd/src/system/classes/exception.php Unable to open file pwd/src/system/classes/arrayaccess.php Unable to open file pwd/src/system/classes/iterator.php Unable to open file pwd/src/system/classes/reflection.php Unable to open file pwd/src/system/classes/splobjectstorage.php Unable to open file pwd/src/system/classes/directory.php Unable to open file pwd/src/system/classes/splfile.php Unable to open file pwd/src/system/classes/debugger.php Unable to open file pwd/src/system/classes/xhprof.php Unable to open file pwd/src/system/classes/directoryiterator.php Unable to open file pwd/src/system/classes/soapfault.php Unable to open file pwd/src/system/classes/fbmysqllexer.php Unable to open file pwd/src/system/globals/symbols.php Unable to open file pwd/src/system/globals/constants.php parsing inputs... Unable to stat file Index.php parsing inputs took 0'00" (2507 us) wall time pre-optimizing... pre-optimizing took 0'00" (5 us) wall time analyze includes... analyze includes took 0'00" (1 us) wall time inferring types... inferring types took 0'00" (4 us) wall time post-optimizing... post-optimizing took 0'00" (701 us) wall time creating binary HHBC files... creating binary HHBC files took 0'00" (84 us) wall time all files saved in /tmp/hphp_3wwUN3 ... running hphp took 0'00" (56575 us) wall time
  • Kendrik: Does this mean that APC is either not used with HHVM now or is redundant. Is there applications of HHVM where it is useful, i.e. where the php files are changing regularly?
  • Markus: the builtin server is not meant for production use, therefore a performance comparison may not be fruitful
  • Edward Fish: The link to specification (https://github.com/facebook/hiphop-php/wiki) does not go to an actual specification.
  • Paul Tarjan: Fixed, thanks.
  • Rudolf Mency: Nice blog and I really liked it
  • 更改 HHVM 代码缓存库路径 | Qing's Blog: […] Go Faster! […]
  • matrix: I have a question. – the first reqeust – http://www.mysite.com/test.php 1. load a bytecode from disk to memory(/path/to/hhvm.hhbc) 2. convert from bytecode to native code by JIT compiler 3. executing native code and cache native code. – the second reqeust – http://www.mysite.com/test.php directly execute native machine code in memory. – the third reqeust – http://www.mysite.com/test.php directly execute native machine code in memory. Is this exactly right ?
  • Nils: I tried the following instructions on my website, though I only got the following error message "Unable to stat file" for most of the php files in my folder. Anyone had a similar problem with a solution? Thanks
  • dev: according to hhvm 3.4, the parameter should be --output-dir /tmp/ , the article may also need an update.
  • Alexander Mashin: 1) Is i possible to set it up more APC-like, that is, to compile bytecode, when it is not present, instead of 404? 2) Would it be enough to restart HHVM service to purge the bytecode cache?
  • Alexander Mashin: Also, 3) my HHVM installation wouldn't start (silently as usual) if I add hhvm.repo.authoritative = true to /etc/hhvm/server.ini. Perhaps, I need some other directives to make it work? And 4) I found a page that expains how to build a repo, so simply restarting HHVM isn't enough. Unfortunately, it seems to demand to list all .php files, and there hundreds of them (it's a MediaWiki). Is there a way to build a repo for all .php files in a directory recursively?
  • Josh Watzman: The article and instructions here are two and a half years old -- a lot has changed since then! Take a look at https://github.com/facebook/hhvm/wiki/Performance-Tuning#use-repoauthoritative---20, which are the modern directions (and are kept up to date) and file an issue on github if they don't work.
  • Alexander Mashin: Thank you.
  • PHP 5.5 vs HHVM: […] did some googling and found the post Go Faster! on the HHVM blog, but it seems out of date. Since then they moved config to ini files which I was […]