Icinga2 graph data retention

Out with Nagios, in with icinga2

Note: I’ve updated this post since I originally posted it.  See my post about xFilesFactor here which describes why I updated it.

I’ve been on a bit of a journey over the last few months migrating from nagios to icinga2 for my systems, some customer stuff and the platforms at my day job after living through the love-hate that is nagios on and off for quite a few years now.

Out with nagios, in with icinga2
Icinga2 is so much nicer than nagios once set up.

On top of the base icinga2 / icingaweb2 install I added the graphite graphing module which essentially worked out of the box give or take a couple of typos on my part .  You can find all the information you need for that part of the journey on the github page for the module.

The graphite module in the context of icinga2 has three moving parts:

  • Carbon-cache, which receives the information from icinga2 and writes it out to disk as efficiently as possible
  • Whisper which is the database format used to store the metrics which can be thought of as a more modern and efficient version of the venerable RRD which all sysadmins since Noah’s time probably have some sort of emotional response to.
  • Graphite which renders the metrics into pretty graphs into PNG format graphs.

The problem, only a day of data

A few days back I discovered that despite a my fairly uneventful installation of the graphite module on my part the graphs, despite being very pretty were only keeping a day worth of data.

Continue reading

When the Cloud goes bad

Before you read too much further this is a bit of a self-serving rant, bought on indirectly by the heartbleed bug published a couple of weeks ago.

I’m a fan of cloud hosted virtual Linux servers for hosting and messing about of the geeky kind.

The ability to spin up a box and play with it to your hearts content then just shut it off again for cents an hour has to be one of the greatest enablers of growth in online services since caffeine laden fizzy drinks hit the market.

I evangelised the use of cloud boxes for web hosting a while ago with some of my geeky friends, which of course rubbed off onto some of the not-so-geeky ones as well.

That’s where the cloud goes bad, right there, just at the end of that last sentence.

I even went as far as to help a couple of the not-so-geeky folks set up cloud servers and migrate their websites in a flurry of command line goodness with bash scripting that is in fact a native language for sandal wearers in many countries.

That was 2009 and skipping forward five years I now seem to have become the Linux agony aunt for a few of the converts and even some of the ones I considered to have strong bash-foo had dug a hole so deep that apt-get and yum can’t rescue them.

One whole day, almost to the minute, after CVE-2014-0160 was published and about four hours after I’d finished the initial patching of all *ix boxes for work my inbox was showing signs of the darker side of the cloud.

The questions were varied, to be fair, but all symptoms of the same underlying issue. “Is my server vulnerable?”, “My server wont update, how do I upgrade it?”, “I googled it but apt-get gives me xxxx error”.

Some of these machines had not been updated by their well meaning owners since they were spun up, in some cases over five years ago. The distribution in the case of Debian 5 now unsupported. This is my problem how?

The next morning I was due to start a three day holiday in the form of being parent helper on a school camp so I really didn’t want to be a part of someone else’s IT dilemma. For work I would keep an eye on things while away, but the other folks? Hmmm.

I replied with a pretty generic message to all of them, along the lines of ‘Patch OpenSSL, replace your SSH keys and if you’re using https get the certificate re-issued’.

So, instead of just relying on the trusty iPhone to keep up I ignored the rules for school camps and packed my netbook along with my toothbrush, socks and bug repellent. My plan being to get online tethered to the phone at least a couple of times during the break.

Well, you could have knocked me down with a feather. About midnight on the Wednesday I get online and there’s about forty emails waiting for my weary eyes.

Half of them unhappy replies to my apparently useless advice and the other half from more people who clearly shouldn’t run public facing servers on the internet without first putting on their overcoats.

Where I had credentials I made a half-hearted effort to get on and update OpenSSL in the wee hours between mountain biking, making hundreds of filled rolls and hut building in the rain but generally I just deffered them as my enthusiasm for cloud computing was being sorely tested. I’m not a very good agony aunt it seems.

And here’s the rant, in a nutshell:

If you don’t know how to keep a server secure and up to date you should not be running your own virtual servers online. Windows or Linux I don’t care. Just don’t do it.

If the concept of migrating applications or websites between releases of Linux Distro is foreign, or you think a public key is something used by kids to sneak into the school pool, get yourself some cheap shared hosting. Virtual servers are not for you.

If you think the scope for a Windows firewall is a tube with mirrors at each end allowing you to peer over the wall…. Well, you get the idea.

I hereby retract all of my former enthusiasm for cloud hosted virtual Linux servers.


The evil Super Admin Password

So you’ve survived a disaster, fire or other adverse event, and you need to shift staff home to work because the office is a pile of smoking rubble. Their PC’s from work are by a stroke of luck usable, and they’ve got broadband. Two thumbs up there.

But about that printer driver you need… It requires admin rights. The domain controller, well it’s at the bottom of a crack in the earth, or in the IT guys garage.

No problem, log in as Administrator, give the local user admin rights, and you’re in business. Oh, they’re an hours drive away, and you didn’t have the fore-sight to install and test a remote control tool.

This is about where you discover why having a single administrator password that is re-used for multiple purposes in the business is considered poor practice. Or, in layman’s terms: down-right silly.

To get the accounts clerk printing, and the receptionist able to configure the network card you’ve now got to give away your precious uber-password over the phone. The kitchen staff can now access skype, but they can also access your bank accounts, the encryption keys for your VPN, the payroll system and the cleverly protected documents with the formula for your world beating popcorn recipe.

You know they will write it on a post-it note and stick it to the fridge, but it beats driving 50k’s across town to fix a 5 second problem… Deal with the fall out later.

So, how many places do you re-use the same passwords? And after the last major outage, did your IT staff have to give it up to the cleaner so he could access Ebay and not tell anyone for fear of having to change the uber-password in 300 hundred different places?

This is part of a series of articles that have come about from my experience in shifting the IT operations for a business after the recent destructive earthquake in Christchurch, New Zealand.

Telecom New Zealand DNS Fail

This tickled my fancy, so just had to write a up a few words about it.

Telecom is the largest telco in New Zealand, and it appears they can’t run a robust DNS setup for their own domain. Their ISP, Xtra, had some problems last year with their DNS, which caused problems for many of it’s customers, but this time around it’s their own corporate domain, telecom.co.nz that has fallen into the cyber bit-bucket.

I was looking for a bit of info on mobile plans, as you do, and went to surf to the site and got a timeout. Odd I thought but it might be my connection so I lept on the VPN to work. Nope, same result. Log into a cloud server in the US. Nup. No DNS entries.

A quick look at the dnc website at www.dnc.org.nz tells me that they’ve paid their bill, so my cunning plan of registering their domain under my name if they’d let it lapse was dashed. However the odd thing is that the largest telco in New Zealand appears to be running their two DNS servers on the same subnet.

100707-telecom-name-serversThat’s not such an issue unless that subnet is out. Which it appears to be right now. Neither DNS server is pingable, which might be the normal state of affairs, but ‘dig’ can’t talk to them either, which is an indication that not all is right in the world of Telecom DNS servers.

So, Mr Telecom, might be time to get with the rest of the world and host a secondary DNS with another provider. Telstra or Vodafone maybe?

100707-digging-telecom-dnsAlternately, I could run one for you in the US for a few bucks a month, it’d save quite a bit of egg on face me thinks.

Taking joy from simple news: IE6 and Youtube

Anyone who has anything even remotely to do with web development will be smiling at the news that Youtube is going to discontinue support for IE6.

Not only that, we’ve got a date. 13th of March, 2010.

While this isn’t really the end, it will certainly put that little bit more pressure on the roughly 15-20% of internet users who still cling to the 9 year old version of Internet Explorer for various reasons I fail to fully comprehend.

You can read more about this on mashable.com or techcrunch.com as they do a much better job than me of writing about such things.

There’s even a response from Microsoft if you so wish to expand your mind.

Having worked in a corporate IT environment I fail to see how even the most lethargic of firms could take 5 years to update the web browser in the modern business environment. Unless you’re talking line of business PC’s in a secure network, but then those PC’s shouldn’t be afflicting their attempts at html rendering on the web development community.

I thought when Facebook stopped explicitly supporting the nearly decade old browser in 2008 that we’d seen then end of it. Then Microsoft shattered the hopes of many geeks, confirming support would continue into 2014.

With Youtube being the number 3 site on the web I’m going to take a punt and say that at least some of the 15% will be getting the message loud and clear soon that it’s time to update their PC.

Security in the cloud, KISS

The idea of keeping things simple when it comes to server security is not at all radical and cloud servers provide the ability to reach the not so lofty goal of keeping your servers simple and secure without breaking the bank.

The theory is simple: The smaller the number of processes you have running on your box the less there is to go wrong, or attack. This is one area where Windows based servers are immediately at a disadvantage over a *ix server, but I digress.

When I was pretending to be a hosting provider a few years ago I ran colocated discrete servers. They weren’t cheap to own or run, not by a long shot. That cost was a huge enemy of the KISS security concept.

In the process of trying to squeeze every last cent of value from the boxes I overloaded them with every obscure daemon and process I could think of. Subsequently the configuration of the servers became complex and difficult to manage, while applying patches became a cause of sleepless nights and caffeine abuse.

With the cost to deliver a virtual server in the cents per hour and the ability to build a new server in a matter of minutes the barrier to building complex applications with a robust security architecture is all but vanished.

The mySQL server behind this blog site is a base install of Debian Lenny with mySQL, nullmailer, knockd and an iptables firewall script. That’s it. Simple to build, simple to configure, simple to backup and simple to manage. KISS.

A little bit of searching around on hardening up a linux box and you’ll quickly find information on changing default settings for sshd and iptables rulesets which you can combine with small targeted cloud servers to reduce the sleepless nights.

I can’t help with the coffee addiction though, I’m still trying to kick that habit myself!

Working on a cloud

This blog is now coming to you from a cloud. A rackspace cloud server that is. Two of them in fact, the front end server running the CMS, and the back-end MySQL server.

The concept of cloud computing really isn’t all that new, but if you’re all at sea when it comes to clouds you might want to toodle over to Wikipedia and read about it there.

“This is the pointy end of the geek scale where crontabs are complex and the preferred editors have two letter names.”
The service I’m using is probably better described as cloud provisioning, in that I’ve got two virtual servers living somewhere in the bowels of the Rackspace data centre. I don’t have to care about memory sizing, disk space, network infrastructure, or anything else for that matter, I’m just renting some resources out of the cloud.

I picked how much memory and disk space I wanted in a few clicks then before the kettle had time to boil the server was on line and ready for configuration. If this service was available back when I was running a hosting business I’d probably still be running a hosting business, although I’d also be stark raving bonkers.

At this point I should say that I’m talking about virtual Linux servers here, not cloud hosting or full service shared hosting. This is the pointy end of the geek scale where crontabs are complex and the preferred editors have two letter names.

I’ve moved the blog onto the fluffy stuff to get a feeling for the service before I shift my work-in-progress link shrinker into the cloud as well. What I want to achieve with the lngz.org is simply not possible on a shared platform, as I want to build a tiered application which can scale quickly.

The traditional way of achieving this goal would be to slap your gold card down on the counter of a hosting company and then proceed to the bank to arrange a second mortgage on your house. Virtualised ‘cloud’ server services such as rackspace cloud, Amazon EC2 or gogrid lets you do the same things for a fraction of the cost and with amazing flexibility.

note: I’m not affiliated with Rackspace, I just think they provide a nifty service. 🙂