A secure Linux server

Ask for help regarding any technical issue or report any bug or OS independent issues.
Post Reply
Frost
TMW Adviser
TMW Adviser
Posts: 851
Joined: 09 Sep 2010, 06:20
Location: California, USA

A secure Linux server

Post by Frost »

I've been having fun trying to set up a secure Linux server to run a TMW-eA (tA) server. Perhaps this will be of interest or amusement to others.
As always, I welcome feedback.


I started with a few assumptions:
- I can stop some people, but not all.
- the crooks are smarter than I am.
- when they break in, they won't just mess up the game, they'll try to steal my data and get me in trouble with my internet provider or the police.

With those ideas in mind, I installed a dedicated Linux system to run the game server.

I manually disabled all network services except the TMW server and SSH (for shell logins). SSH is running on an alternate port and I've configured additional security options for it.

I set up a separate firewall in front of this server, effectively putting it in a DMZ. The firewall blocks any incoming traffic that I don't expect (so a crook couldn't install an FTP server here) and allows only the minimum of outbound traffic. Connections are logged. Even if someone takes total control of the server, they still can't change this firewall or attack my home network. Of course, I still configured iptables on the server too.

The tA server runs as a normal user, not root. I disabled interactive logins for that user account.


Still to do:
- try to put tA in a chroot jail
- move the tA server and data to a separate filesystem, then mount that filesystem nodev, nosuid, and whatever
- send system and tA logs to a separate loghost
- set up automatic backups (read-only, of course)
- configure SELinux policies and turn on enforcement
- set up Nagios monitoring
- change sshd to use either a One Time Password system (eg. opie or s/key) or X.509 certificates
- install and configure Tripwire
- find a use for it ;)
You earn respect by how you live, not by what you demand.
-unknown
User avatar
Crush
TMW Adviser
TMW Adviser
Posts: 8046
Joined: 25 Aug 2005, 16:08
Location: Germany

Re: A secure Linux server

Post by Crush »

When you set up an eAthena server you are very likely to also install some other applications like a webserver for updates, a website for your game and maybe some other stuff like an IRC server, git repository and what not.

The most intuitive way would be to install all these applications on the same system. But this means that a security vulnerability in one of these applicatons can also take down all the other. Just one example of many: when you create a web application it is very easy to accidently code a directory traversal security hole which exposes the whole filesystem of the server to the web. When you have just the website on the host, that might be unpleasant but not necessarily a catastrophe. But just imagine what would happen when one can also read you eA account database that way?

For that reason every application should be installed on a dedicated system with the bare minimum of software around it to support this application. What? I am crazy? No one can afford so much server hardware? You are wrong. You don't need more than one physical server. The keyword is virtualization.

Virtualization is a technique which allows you to simulate virtual computers inside another computer. You can create a virtual machine and then install any operating system on it. You can put the virtual machines into a virtual network with an internet connection provided by the host system and then interact with them like you would with any other system. The software running in these machines doesn't even notice that it runs in a virtual environment. It works like it runs on a real system, but there is no way to access the host system or the other virtual machines in any way.

This means when someone compromises your webserver, gets root access and wreacks havoc with it, there is simply no way to access the gameserver even though it runs on the same physical hardware.

Needless to say that all virtual machines should use different passwords.

Virtualization also has some additional benefits, like being able to create snapshots of servers and restore them later, clone a virtual machine to conveniently create a test environment identical to the production environment or transfering a server from one hardware to another with a simple copy operation (set it up on your own workstation, when it runs just copy the virtual machine to the server and it keeps running as if nothing happened). The cost is, of course, some performance overhead. At the company I work for we have a rule of thumb of 10% cpu and memory.

Speaking of work: we currently have over 100 physical servers in our datacenter which run over 1000 virtual servers. By using virtualization we were able to reduce the number of physical servers by 50%, keep the workload of our system administrators constant and improve our uptime statistics by an order of magnitude, although the number of applications which are hosted in our datacenter grew a lot in the past few years.
  • former Manasource Programmer
  • former TMW Pixel artist
  • NOT a game master

Please do not send me any inquiries regarding player accounts on TMW.


You might have heard a certain rumor about me. This rumor is completely false. You might also have heard the other rumor about me. This rumor is 100% accurate.
Frost
TMW Adviser
TMW Adviser
Posts: 851
Joined: 09 Sep 2010, 06:20
Location: California, USA

Re: A secure Linux server

Post by Frost »

Crush, you're right that virtualization can be a great way to segregate services from each other without having to buy more hardware. On UNIX-like operating systems, using chroot can often achieve comparable results without the overhead of virtualization or having to manage multiple hosts.

(That said, I first attempted to install tA in a Solaris zone for just the reasons you mention. I never got it working, though.)
You earn respect by how you live, not by what you demand.
-unknown
Matt
Grand Knight
Grand Knight
Posts: 1759
Joined: 07 Aug 2004, 10:47
Location: Germany->Bavaria

Re: A secure Linux server

Post by Matt »

I too would favor chroot over virtualisation for this job, but Crush seems to be collecting vmware points, working for a vmware partner or did just spend too much time with the vmware salesman :D
Frost
TMW Adviser
TMW Adviser
Posts: 851
Joined: 09 Sep 2010, 06:20
Location: California, USA

Re: A secure Linux server

Post by Frost »

Matt wrote:I too would favor chroot over virtualisation for this job, but Crush seems to be collecting vmware points, working for a vmware partner or did just spend too much time with the vmware salesman :D
;)

Yeah, I have a bit of experience with VMWare and Solaris zones/containers in small (1 rack) to large (500+ racks) sites. It solves some problems, creates others, and isn't really a security solution. That could be another thread. :)
You earn respect by how you live, not by what you demand.
-unknown
User avatar
Hello=)
TMW Classic
TMW Classic
Posts: 656
Joined: 11 Jun 2009, 12:46

Re: A secure Linux server

Post by Hello=) »

- try to put tA in a chroot jail
If you have more or less recent Linux kernel, there is a much better option exists: LXC (Linux Containers). In fact, it's a matter of one system call to start new process in a isolated area (and there are utilities to do this in convenient way). You can have separate filesystem namespace, separate processes list, etc. In a very recent kernels, containers are even able to use a "fake root" accounts (i.e. it appears as if you were root, but this "root" is only able to manage things in own container). So you can manage it as if it was separate virtual machine, if you want to. Or you can start just isolated process. Parts of this technology existed for ages in so called OpenVZ patches (used by dozens of VDS hostings). These days most interesting parts are in mainline kernel, so you don't need to bother using custom patched kernels: if you have recent kernel, it's already here. Btw, Google uses this technique to secure Chromium under Linux. You can see their wrapper source if you're interested: it's quite small and clean thing and definitely adds up to their defence. And it's much more secured than chroot.

Pros of LXCs:
+ Much better isolation than dumb chroot. Probably even better than SELinux (which is being disabled by each and every public kernel exploit anyway). Chroot was never designed as security feature so it may allow certain ways to break out of jail, especially if attacker has managed to elevate himself to root.
+ No speed overhead. Unlike with a fully blown hypervisor which is doing a full virtualisation of hardware (vmware, xen, kvm, ...), there is very little overhead and speed you will get is virtually same as if you were running directly on "host". So you can get a bare metal speed while be able to use some virtualisation-like features and enjoy by decent processes separation (you can even limit resources separately). The worst case speed penalty is just 2-5% or so. So you would not notice it at all.
+ Not so hard to setup. Modern kernels are already coming with it, so it's a matter of installing few tools (usually from default OS repos). IMO it's a way more fun that dealing with a crappy SELinux policies etc and leads to even better result: we instruct kernel to keep certain processes isolated from other parts of OS in a quite efficient and predictable way which is IMO easier to supervise than numerous ACLs and policies (which are hard to keep in brain so there is risk of mistakes).
+ You can manage it either as fully blown OS installation (up to having own IP for that "guest os", root account and custom set of packages) or just use it as advanced chroot-like approach. managing things from host.
+ You can control how many resources all this could consume.
+ In fact, there is only 1 copy of OS kernel booted for all. So memory overhead is not as high as with full hypervisors running truly separate copies of OSes and their kernels.
+ Minimalistic setup which only runs one or few services is easy to monitor for changes. If you're paranoid, you can try to put web server and eA into different containers, leaving communications between 'em via network only and don't leave access to host IP at all from their side. Since overhead is quite low, it's affordable. It would not work out of the box but shouldn't be a rocket science either.
+ It's easy to place some traps in containers without chance to suffer yourself, since you can use cheat: host could access all resources and you can do your jobs from the host side ("supergod mode").
+ You may use checkpoints for fun and profit.

Cons of LXCs:
- Still, you have to spent some time and effort to configure it and get familiar with this technology.
- Some things will not run out of the box, but as long as you're familiar with chroots and overall system management, that shouldn't be a serious problem.
- You can't use "level 5 system magic" known as "live migration" yet. Doing so would require either fully blown hypervisor or at least messing up with OpenVZ kernels.
- Still, full hypervisor which virtualises hardware would offer somewhat better protection (at the cost of speed). Say, in Xen it's most critical key part, the hypervisor who manages resources access and who haves total access to everything is quite small in terms of code size. This means it's very likely there are fewer bugs in xen's hypervisor than in something larger like Linux kernel. Sure, there are still chances there will be bugs (and they were found in the past). But the smaller the code, the fewer bugs you could expect. Disadvantage is obvious: full hypervisors imply serious speed penalty. You can get 90% of computational speed of CPU quite easily. But when it comes to massive IO, full hypervisors may deliver you some headache. Disk I/O or networking could cause much load on host's CPU and could be much slower than "bare metal" I/O. And servers are about to making lots of I/O, unfortunately.

Some hints if you will try virtualization or containers:
1) Never allow administrative network access to "host" itself (physical machine running containers/VMs) to anyone but yourself. Assign separate IPs to VMs/containers running guest and make only them available into public internet, not host itself :D. This may require certain port forwarding, NATing and firewalling, depending on how many routable IPs you can afford and what's the setup.. Once host hacked, every guest on it could be hacked and you have a problem. On other hand, if host is not accessible via network, there is almost nothing to hack. And hacked container/VM isn't a really huge issue. At least it's much easier and faster to re-create container/VM or rollback to clean good snapshot than perform full OS reinstall on real hardware. And it's easier to detect :mrgreen:.
2) I believe the best defence is a lightning bolt, popping out of nowhere at enemy head :mrgreen: . Have you heard about rootkits? With virtual techs you can make similar approach working on your side, striking hackers from nowhere when they do not expect it. Look, hacker can't see host processes. And full filesystem. But you can monitor all container activity in invisible way from the host. You can check say, md5 sums of all container's files in background or catch all FS accesses in your monitor, keeping checksums, logs and monitors outside of container view. Hacker would neither see monitor(s) in processes lists nor would be able to kill them it or erase logs. As for me, it's funny enough.
3) Hey, have I told that host could conduct administrative actions on guests from it's side? It can access all files from his side, inject new processes onto guests, etc. This allows to make some key system utilities in guest containers useless stubs who rather serves as trap. You want ls? Uname? Something else? Wow, cool. But under normal scenario web server's process does not needs them, isn't it? And legitimate admin could do ls on full FS view from the host side. So if we would replace ls and uname with trap which signals to host that container is compromised (then host may shutdown container and send alarm), we can have some fun, looking how hackers are reporting self and starting shutdown sequence (maybe with checkpointing, to study system state and hacker's actions later). And the best of all, hackers can't easily harm all these sequences as host activity is invisible for them. They would only notice that something went wrong only when container is shut down, admin alerted and it's too late. I'm thinking it's funny enough to use their most advanced techniques similar to "rootkits" against 'em :wink:.

Note: that's only my own view and since I deal with virtual environments I can imagine some interesting ways to use them. It does not provides perfect security but seriously raises the bar.

The overall idea is: imagine that you're hacker. Imagine that you want to hack that host. Then imagine your worst fears and worst surprises you could expect on the way. Then implement them and it will be a good chance that hackers would be unhappy to eat this as well :twisted:.
Frost
TMW Adviser
TMW Adviser
Posts: 851
Joined: 09 Sep 2010, 06:20
Location: California, USA

Re: A secure Linux server

Post by Frost »

t3st3r, thanks for the detailed analysis of containers. I've used LPARS and Solaris containers, but didn't know this capability existed in noncommercial Linux. I'll take a look.

One caveat to virtualization as a security tool: the most recent security models I've learned involve classifying data into security categories, then segregating at the data level rather than the OS or process level. In other words, if you store sensitive information (eg credit card numbers) on a physical disk, then you apply that security requirement (which in the financial industry involves regular external audits) to any device that can access that physical disk. It's a game of cat-and-mouse with techniques like virtualization and file-level or device-level access control lists, but thus far there's no exploit for the old-fashioned "air gap." ;)
In short, in that model you wouldn't put your anonymous FTP server on the same hardware as your back-office data processing server.

P.S. I've heard of rootkits...and bluepill.
You earn respect by how you live, not by what you demand.
-unknown
Post Reply