March 2006 - Posts
Like many people, I read the
original article on el reg yesterday and
the
conversation log
it linked to. Frankly the majority of the blame
attaches to the end user, but nobody covered themselves in glory there.
Reading the exchange of emails shows plenty of times where either party
could have easily stopped the discussion before it sunk to its eventual
depths, and instead a sarcastic reply or a threat was chosen instead.
Real smooth.
The register has a
further piece
today on some of the feedback the original article generated, too.
Needless to say, another paragon of reasoned and balanced discussion
managed to carry the news too, with
predicatable results.
While reading Joel On Software's message boards today, a comment in a
thread there caught my eye:
"Oh my. Now he's been fed to a baying mob of indignant socially inept
geeks. "How dare he not understand? How dare he attack intellectual
sensibilities? How can ANYONE be so ignorant? What a dumbass"
[...]
This is turning into a religious battle with all the venom and
bitterness of a real life entrenched struggle. Where are the people who
bridge the gap, the peacemakers? Oh they're being drowned out by the
baying geeks. We've seen several instances recently of mob rule on the
internet - and it's all been pretty ugly."
I'm disappointed, but not surprised, by the "baying mob". This always seems to
happen in IT discussions. If this mob were representative of the IT industry then
they're a shocking indictment on how unprofessional and immature our industry really is.
Hopefully, these mob are not representative of the industry, but are
almost certainly being seen as if they are by outsiders
. The end user made a mistake,
got
punished for it, got
mocked for it a little bit. Fine. OK, he
was a ass
and deserved to feel a bit of pain for that. Perhaps.
But there comes a
point where it crosses over the line and the feeding frenzy of people
looking to land their own little personal kick starts to become
disturbing.
If you were part of the
crowd
who rushed to flame this person, I have a question for you. If you feel
ill and get taken to the hospital and say something there which sounds
silly to a professional medical carer, would you be ok with every
doctor and nurse trooping into your room to make fun of you for it
constantly for a month? No. I didn't think so.
Something to think about before people rush to flame the next person to make a mistake eh?
One more thing. When we do have problems with systems, we expect users
to tell us. Events like this do nothing more than teach end users that
there is no point speaking to the IT Experts because they'll either get
patronised or outright made fun of. And that helps no-one.
One of the more common areas of confusion with Virtual Servers is how to deploy them properly, to ensure good performance and reliability. Some people are scared of Virtual Server technology and refuse to believe it can ever perform well enough to justify the investment. Others see Virtualisation as the magic bullet and end up throwing lots of money at technology they don’t really understand, with disappointing results.
While the Microsoft
Virtual Server product is quite new, Virtualisation is a mature technology in general, with
Virtual PC being a long established product and
VMWare having a large range of virtualisation products to fit most needs and budgets.
When deciding whether or not to use virtualisation in a data centre, you should first of all formulate a list of aims you expect your virtualisation project to achieve. This should be painfully obvious but I’ve seen lots of IT projects implemented because “it looked cool” and these are usually characterised as the ones that end badly and cost way over their original budget.
Three of the most frequently expressed motives for a virtualisation project might look something like this:
Migration of old legacy servers onto new hardware. This frequently reflects old “line of business” apps running on NT4 or old versions of Linux that cannot easily be upgraded to more modern OSes and where maintenance on the current hardware platform has become an issue.
These deployments are usually pretty painless because the Virtualisation process is simply providing a compatibility layer to allow an old OS to run on new hardware for which it would not normally have drivers.
These old server applications typically will not stretch the abilities of modern hardware, so you can probably get away with sticking two or three applications of this kind together on one system without too much thought and get away with it. The use of the word "probably" is important here. Test, don't assume!
The Middle step of a “Swing Migration”.Where users want to upgrade a server in place, but feel a clean install of components is either “required” or at least preferred, then virtualisation can make sense to reduce the hardware requirements for the temporary server used to hold the data while the “proper” hardware is being upgraded.
This hasn’t really come into play a lot so far, but right now we’re seeing people buy “64 bit ready” hardware and running 32 bit versions of operating systems and applications on it while they wait for 64 bit versions of those apps to appear and grow in maturity. This is a common scenario for people who are using SQL Server or Exchange Server on Windows at the moment… 64 bit versions of Exchange are still in beta at the time of writing, and 64 bit versions of SQL Server 2005 are available but very new, and all good DB admins are cautious about doing too many new things at once to data they actually care about.
Once the tipping point arrives for these systems, it will not be possible to upgrade in place and setting up a temporary system on a virtual server to move the “live” install onto while the proper host system is being rebuilt makes a lot of sense.
In a way, I see the performance issues for this situation being quite similar to the issues you would encounter with moving a legacy system. You need to do some planning of course but you probably can just get away with “throwing” the system onto your virtual server host any way you can, because with luck it won’t be there for very long. You can also limit the amount of systems to be virtualised at any one time to whatever capacity your virtual host can cope with.
Consolidating server hardware.This is usually an attempt to reduce the investment in hardware that would otherwise be required in a data centre. Where you have several processes that don’t consume many resources in themselves, it is always tempting to combine them to reduce your hardware outlay. But this presents a number of problems:
A planned interruption to apply a patch to one service may require an interruption to all services if a reboot is required. Not only are un-needed interruptions to services obviously undesirable, but schedules for planned work are instantly made much more complex.
Change and Release management become much more complex when several services run on the same system and possibly share the same system components (for example, Java Runtime Engine or .Net framework). What do you do when you have two components on the same system using the same framework, with one validated for, and needing, an upgrade and the other lagging behind on this approval?
It should be clear that all of these scenarios require at least some planning, with the first one I present needing the least planning and the final scenario needing the most planning - how much exactly depending on what kind of deployment you ultimately sketch out.
Some parts of the planning are pretty much settled for you from the start. The more RAM the better, based on what the host platform can physically support and what the guests sharing it require. With the obvious caveat that you probably don’t need to assign 4GB to a guest that only runs a dedicated DNS server, for example, RAM is always a no-brainer.
CPUs again are pretty much settled for you based on what the host will physically support and how your virtualisation software will allow you to allocate physical CPUs to guest machines. This again is a no-brainer, if only because your options here are only limited to a few.
I personally suggest using at least a dual-processor machine for your Virtual Server host, and making sure you work on utilisation figures somewhere between “average” and “worst case” when calculating CPU loading for each guest in order to minimise the amount of contention between virtual machines sharing the same CPU. For tasks that can run processor intensive I suggest on planning on one physical CPU to one virtual machine.
An area where planning can really make a difference are the virtual disks. On a poorly put together system it isn’t uncommon to see virtual disks from several guest systems thrown onto a hard disk with little or no thought. This will cause immense performance problems as the systems grow in size and complexity due to the large number of reads and writes being made to the same hard disk from different processes.
There appears to be a perception that
SANs are magic that is perpetuated by some SAN sales teams. Just buy a few disks and throw your storage needs onto them as you will, goes the mantra, and all will be fine because of our magic caching system. The same thoughts seem to pervade virtual machine disk planning, and are equally false wherever you hear them.
Actually I think it helps to use the same sorts of tools and thinking for good virtual machine deployment as it does for a good SAN deployment. In both cases, the physical location of the data stored is “abstracted” from the server using it in some way, and in both cases we will tend to see several different servers using the same storage hardware.
So, take the "virtual" (or “SAN”) out of the equation for the moment. What matters is the kind of server app you're trying to deploy and the use it is going to get (testing, production use, DR recovery simulation, etc).
You've got some data and you've got some disks. Now we're just doing standard server deployment planning; there are advantages in performance and reliability in placing different parts of a server application on different disks. If it makes sense to put the OS, transaction logs and main database files onto different physical disk spindles in a physical SQL or Exchange server, it makes just as much sense on a virtual server that is being deployed to replace that physical server.
So the basic rules might be: (note that most of these answers are suffixed with a silent “unless you’re using a test system where performance is not an issue”).
- NEVER store virtual disks on the same spindles as the host operating system.
- NEVER use software RAID on a host server!
- NEVER use software RAID on a ‘production’ guest server
- Run - don't walk - away from any vendor who suggests throwing it all onto the largest hardware / SAN RAID 5 that you can afford and letting God sort it out. They're either incompetent or actively out to screw you.
- Think very carefully before you place virtual disks from different guests on the same spindle. I don't want to say "Never" because the actual impact depends on your pattern of use for these virtual disks. Again: Test, don't assume!
- When deploying virtual disks, the same rules apply as they would for the same process on physical disks. For example, do not allow a database store to use the same disk spindles as its transaction logs.
- Consider how you will back up each server. Be especially wary of “magic” backup processes that work at the host server level, and test to ensure that you can restore your guest servers to working condition.
- If two or more servers are designed to provide redundancy for each other (e.g. domain controllers, primary and secondary DNS, etc) then NEVER place them on the same host machine!
- If copying virtual machine images to “rapid deploy” servers, then be very careful about things like duplicating SIDs. I know it’s boring and adds to the deployment time, but I still suggest using sysprep for these kinds of images.
With the
baseline install image ready to be produced, you should tidy up OS components if you haven't already, and create a new user to use as your "template" for the default login environment. This account should be a normal user level account rather than an admin account or anything else unusual.
Ensure that all apps run under this user account. In fact I suggest you make sure that you test all your applications once more, to catch things like "per user" registration details. Configure default settings for applications, etc. from this account. This is your chance to insert any "standard" templates and make site specific changes to applications (for example, we enter our 'standard' Final Cut Pro settings at this point).
Arrange the Dock how you want, and ensure any custom user settings such as wallpaper, screensavers, accessibility settings, are all how you want them to be.
With all this done, logout of the template account, then login again with the root account and copy the contents of the template user’s folder to
/system/library/user template/English.lproj (this folder may be different depending on the language/locality you're working in).
Next, check the copy you just made in the templates folder and remove any settings that you don’t need – pay special attention to things like keychain files, and "bloat" from programs that install lots of default template information into areas such as
~/library when first ran.
Finally, reset permissions using either the disk utility for the whole disk or using the chmod command directly on the
English.lproj folder.
At this point, you're finished but you should test your new template by creating and logging in as several new user accounts to check the template works as expected for a new user. Don’t forget to delete these test users prior to making your final deployment image!
Apple OS X can very easily be cloned to apply to other machines in a procedure vaguely similar to using Symantec Ghost to clone Windows machines.
OS X images are hardware independent so you could create an image on a Mac Mini and deploy it to a Mac notebook and desktop machines without worry. The only time you need to keep multiple images is to support different processor architectures; you obviously cannot expect to deploy a Power PC image to an Intel processor based machine and expect it to work!
- Images can be taken, using several tools, and stored on either a FireWire disk for deployment by hand, or on an OS X server for deployment via NetBoot over the network.
- The package we use is Carbon Copy Cloner which is shareware so can be downloaded and tested for free – and the author generously allows education to continue to use this product for free!
- Also from the same site, DeLocalizer is a useful tool for removing language settings which you don't use from the OS installation, which can save over a Gigabyte in disk space… which really helps when deploying over a network!
- It currently takes about an hour to deploy a 30Gb OS + apps image over our network. This compares well to a network deployment of Windows type machines with our full package load.
I suggest performing a clean install of OS X onto a system to use for your image "baseline". This gives you a chance to ensure that you definitely have all the components you need present, and that you remove all the un-needed stuff that you can. An extra hour or two here can save you
days later on if you make a mistake!
When installing your "baseline" I suggest performing a custom install of OSX and taking the chance to remove things such as printer drivers you know will never be used on your network, and such-like. Some default installs also include application software demos which should also be removed at this point. This can make a surprisingly large difference to the OS install footprint.
If you are using a copy of OS X that came bundled in with a Mac for your baseline image, remember that tools such as
iLife and
iWork are not part of the base OS, but in fact are what Windows users would describe as an "OEM bundle". You cannot install iLife or iWork on a Mac unless you have a licence to do so, so you should leave these out of your baseline image unless you're certain you have the right licences to allow you to make a mass deployment of these packages.
This post is an "extended reply" based on a question and my reply from the Microsoft public newsgroups.Q. I'm not sure if what I understand about malicious network traffic (external/internal) is correct, or maybe this concept is very subjective or complex.This is a very complex area - it can be difficult to identify particular streams of network traffic and match its pattern to something that
may be malicious.
Ah... I'm already writing phrases like "may be". Why do people like me always answer questions like this so vaguely and evasively? The problem of spotting and qualifying network traffic as malicious can also be subjective.
Let me give you an example:
VNC. Fantastic tool that lets you take control of one computer's desktop from another.
I run VNC on my Apple Mac laptop at home so I can control things on the Mac from my windows desktop machine. This isn't malicious; I own both machines and I install VNC knowingly, to help me do what I want to do.
On the other hand, let’s say I send an email to you, with instructions that trick you into installing VNC without telling you what it. Next I proceed to use it to hack into your computer to steal whatever it is you'd really hate for a stranger like me to steal. Now we're being malicious, yet it’s the same bit of code both times. The program code hasn't changed, and neither has the "footprint" of VNC being used on a network.
The code and network traffic are not malicious one moment and fine the next, what has changed is the intent of the person installing and using it. "Malicious" implies intent. A mistake in a configuration setting isn't malicious as there is no intent to cause harm. However, such mistakes will probably make you more vulnerable to an "exploit" - taking advantage of the mistake for malicious ends.
Q. Ok. I've read about network analyzers/monitoring, using sniffers like Ethereal. I also understand that you can sometimes spot malicious traffic in things like ISA server logs BUT once inside of these tools I can`t identify malicious traffic. I have spoke with experts in matter and always they recommend using sniffers and similar tools but to the question "how can I identify malicious traffic once inside of these utilities?" they respond vaguely and evasively. Does this malicious network traffic have some clue (protocol, port, frame, size ...) that can help to identify it?As you hopefully have guessed from the previous part of my answer, there is no simple way to identify malicious traffic because there is no simple definition of what constitutes malicious network traffic.
As my VNC example shows, the same bit of code can be "malicious" on my computer because I didn't put it there and just fine on your computer because you want it there. As I’ve already said, I run VNC server on my Mac so that I can remote-control it from my Windows desktop. However, I don’t want my Windows desktop to have VNC server installed. So you could run a network analyser on my home network and spot VNC, but there is no way to tell the intent behind the use of that tool without understanding how I use the network.
So remember that while you can look at network traffic with a sniffer, you can't know the intent of the user by looking at the network headers. There are some patterns that indicate malicious traffic sure, but most of the time nobody can give you a magic answer and say "this protocol" or "this port" is malicious.
So how would we identify malicious traffic from within a sniffer? Well we probably wouldn't, as it isn't the easiest method. We can use
automated network scanning tools that identify certain kinds of traffic that can ONLY be malicious. These aren't perfect but are useful. We can also use firewalls and other similar tools that are good at spotting malicious looking patterns in network traffic and blocking them. Again, not perfect but very useful.
As for using a sniffer manually, this can help in two ways. If you know your network VERY well then you might notice something unusual via a sniffer. It's all too easy to miss important things by relying on this but I thought I'd mention it. More usefully, once you know through other methods that something "wrong" is going on, and hopefully you've narrowed the problem down to a few machines, you can use a manual sniffer to inspect traffic to/from your suspect machines to get a good picture of what is going on. In other words, manual traffic sniffing is a bad method of detecting a problem, but a good tool to use for investigating a problem once you know it’s there.
Apple have released a large security patch,
2006-001, which addresses a number of issues, including
that one
in Safari and LaunchServices. Quite a diverse range of issues get
tightened down with this patch so as always I suggest immediate testing
and roll-out at your earliest conveniance.
The patch is already available via software update or via the Apple download site
here - be sure to grab the right versions if you go this route!