I’d like to take a moment and talk about scaling intelligently when dealing with Magento in an Enterprise environment. We’ll cover scaling your web servers, database, and administrator properly to avoid some of the headaches involved in running a larger store.
When you start to think about running a website at scale, it’s important to stop and evaluate how you think about an application. One of the primary elements of scale and the modern architecture is being able to treat the different pieces of an application as individual pools of functionality. These pieces are interdependent and should be built redundantly wherever possible.
Scaling the Database
The easiest way to scale Magento is to start with your database as it’s relatively easy to move the database to a dedicated server and massively expand your capacity. Running a database tends to be memory intensive with large IO requirements so ensure you’re on a solution with a lot of RAM and extremely fast(SSD preferred) drives. The processor should be adequate but is the least critical item in a database server build. For a medium to large store, I would suggest a quad core server with at least 16GB/RAM, depending on the size of your database. MySQL, in my experience, will run just fine in a virtual environment which should allow you some capacity to scale vertically. I would like to say that this is the only component I recommend scaling vertically.
Properly tuning your database can be a bit difficult and I would always suggest contacting a qualified DBA to get things set up properly. That being said, MySQLTuner does an excellent job of telling you what to tweak and is something I use regularly. You’ll need to install perl in order to use the tool and will need the MySQL root password. MySQL takes a while to get into what you might consider its “running” state in regards to buffer sizes, etc. so it’s recommended to let your MySQL server run for a while before running the tool. MySQLTuner can be downloaded at http://mysqltuner.com/
Separating the Admin & Scaling Your Web Servers
OK. You’ve got the database separated. What’s next? The next logical step is going to be scaling the front end(web servers) in your environment.
I’m going to assume you’re using Apache for this article though we could look at Nginx/PHP-FPM in a later post. When it comes to scaling web servers, bigger is very seldom better. You don’t really want to shove all your traffic through one big web server, you need to break it down across multiple smaller web servers on the front end. I would suggest making these somewhere in the range of 4 Cores with between 8 & 12 Gigabytes of RAM. For the web server, processor speed is going to be a much bigger part of the equation than RAM or the disk. Keep in mind that we’re discussing a web sever, not a caching server. These are different components.
So, let’s say you’ve got four front end web servers. They’re all writing happily to the same database and accepting visitors successfully. We’ve still got a few problems to deal with.
The first involves the Magento administrator. When you’re working with the administrator, there are certain bits of functionality that involve writing files to disk. This includes uploading products(images), running imports/exports, etc. and these files have to be shared by all servers. My preferred solution is to move the administrator to an external server and configure rsync to sync your Magento installation from the administrator to the required hosts on a 10-15 minute schedule. This allows all file uploads to be functional on the frontend within that 15 minute time frame which is more than acceptable for most enterprise stores. I wouldn’t suggest going lower as that might cause unnecessary strain in some scenarios. It’s also strongly recommended to remove the var folder from this rsync.
The second problem to deal with is cache. By default, Magento stores cache in the var/cache folder. When you clear the cache inside the Magento admin, you’re clearing only the cache for the server the admin happens to be sitting on. In order to clear cache effectively, you can store your cache on a shared mount point(you could have done this instead of rsync as well) or utilize a third party cache server like Memcache to maintain a consistent cache across multiple servers. Here’s a nifty little tutorial on configuring memcache. Just remember to set it up on an external server and point all of your Magento installs to your Memcache server.
When the default Magento cache isn’t quite good enough, it’s time to take a look at setting up a dedicated cache server. Cache servers, like Varnish, are designed to respond very rapidly to static requests while passing dynamics back to a web server, like Apache. I would start with a 1-1 ratio of cache servers to web servers, look at your performance and load metrics, and tweak accordingly. Without getting too deep into the implementation, the plugin below will assist you in configuring Magento to use Varnish. I’ve also included an article below on Varnish configuration.
Check out the next article in this series: Quick & Dirty Automated Deployments with Github, Jenkins, & Magento.