Category Archives: Linux

MySQL Why Won’t you Dump already!

Few weeks ago I started up a project migrating a development mysql to the production environment.

Doing this change included a lot of tests and updates I wanted to do for a long time.
The short story is that I migrated user defined mysql data (permissions and triggers was vetted before) to Master Slave on docker images and a different user with the entire environment configured via Ansible (yes yes I went overboard with the design – but hey, now I have all the templates I need and every docker and mysql needs and documentation and all orchestrated by ansible)

I’ve needed to create a special docker image for MySQL 5.1 for this project – I will post a followup post with the information about the docker

After all the hard work preparation etc I was ready to do the migration! Finally! I’ve dumped the data, Upload it to the new Master and Slave, Changed the users (Create new Delete the old ones),  sync everything and set the backup.

The next day I woke up and checked the backup and found it didn’t work, I was not in a good mood – the backup failed… Continue reading MySQL Why Won’t you Dump already!

SELinux – Why I need it and why is it still on??

If you know CentOS or Redhat you probably familiar with SELinux.
For me the history with SELinux is simple – I know about it – I always disable it – and that’s it.
As probably most of you know there is abundant of explanations on the web on how to disable/enable/configure the selinux – So I won’t hassle you with the information.

However I would like to share a fun story about one of the engineers that tried to disable the feature to activate something and was wondering why the selinux keep starting up when he reboot the server.

In my environments for the past few years I disable the selinux via the pxe+kickstart installation or ansible, but now some of our engineers started sprouting their own centos servers in AWS for testing – and they started having several issues with configuration, Continue reading SELinux – Why I need it and why is it still on??

The Curious Case of Missing Quota Report on Volume

A week ago one of my old colleagues ask some help with an error he has on one of his servers, I was happy to oblige.

He had several servers all working exactly the same, same hardware, same OS and same configuration (apart from IP’s).
One of his server crashed and after the reboot one of his batch scripts stopped working specifically on this server.
The script tried to determind if a specific volume (for example: /mnt/volA) exist and have enougth free space,
This he did by invoking the “df” command and the “quota” command (Or their equivalent in the specific language).

What he found was:
While running: “df -k /mnt/volA” he got the correct line and response,
But on the other hand when he tried to run “quota -v |grep “/mnt/volA” the quota command didn’t return the volume.
This was particulary wierd because this volume was NFS volume with specified quota (set on the NFS server) – moreover the volume did apeer on the other servers when you looked for it when you’ve used “quota” Continue reading The Curious Case of Missing Quota Report on Volume

The Curious of the missing LDAP Groups Issue with new groups in LDAP

LDAP is a wonderful way to handle all your company employees users. I’ve been using it for a long time and just like the easy way to handle it and how users can be authenticated via the LDAP directory all threw our intranet sites (Ticket manager, Wiki, Tools and more).

A month ago I had a new project – simple and easy to create some directories for group of users and give some full permissions and some only read permissions.

Due to the nature of linux that was quite easy – Just create a new group (or several) to encapsulate the users and allow the permissions to the directories. For example:
1. Group: project-users       – All the users who will need to access the data for this project will be added to this group.
2. User: project-admin – This is the user that can handle the directories (add, remove etc).
2. Group: project-admins    – The only group that can access the project-admin users. I’ve added only selected users to this group. (The permissions to access the users is auto generated authorized_keys access)
Now let’s say that our directory is:
/opt/projects/project (User: project-admin, Group: project-users, Mode: rwxr-r—)

Now as you can see only the admin user can change the files but all the group can access.

Continue reading The Curious of the missing LDAP Groups Issue with new groups in LDAP

Apache HTTPD config check – Or – Check that damm engine before I try to fly this bird!

Although large amount of alternative web servers a lot of the web services out there still use Apache httpd software.

I will not go into why choose Apache httpd and how, but I like to share a  small issue I had with configuration check of my service.

As IT personnel who worked in a production environment I’ve learned the best practice of any configuration change is check your configuration before reloading/restarting the service.

Now Apache httpd configuration check is easy:

# apachectrl configtest
---- or ----
# apachectrl -t

Now – What about when you have multiple apache httpd instances running on the same server?
Every one of them running with it’s own configuration file.
How would you test them?

The solution should have been simple enough – just direct the check for the configuration file!
The issue with that is the relative selection of file won’t work.
Lets see a small example:

---- Following test will result in checking of /etc/httpd/conf/httpd.conf instead of what you wanted ----
/etc/httpd-site/# apachectrl -t -f conf/httpd.conf
Syntax OK
---- Following test will fail because he is looking for /etc/http/../conf/httpd.conf ----
/etc/httpd-site/conf.d/# apachectrl -t -f ../conf/httpd.conf
httpd: Could not open configuration file /etc/conf/httpd.conf: No such file or directory

So how can you fix that?

---- Use full path ----
/etc/httpd-site/# apachectrl -t -f /etc/httpd-site/conf/httpd.conf
Syntax OK
---- Set the work dir for the relative search ----
/etc/httpd-site/# apachectrl -t -d /etc/httpd-site -f conf/httpd.conf
Syntax OK

Easy and simple.

Storage handling with users in Linux – or – The Land of Forgotten Inodes

In my years in IT I’ve handled a lot of storage (proprietary and open source solutions).
But one of the main issues that I had (beside design of large storage and backup) is the rouge user/script/program – sometime one of them will act up and fill the storage with large amount of files or one massive file (for example a big log file).

From time to time someone comes to me and say “Well my script isn’t working” – one of the first things I do is running “quota -u <USERNAME>” and 90% of the time We both see that the user filled his size or inodes quota and hist script fail.
From time to time I collected a lot of scripts and one liners to help me search for the problematic dir/file that contain the issue.

Here are some nice one-liners that can help you find the culprit:

Beware! Using some of this one liners on a directory that have under it local or NAS mount (example: nfs) won’t yield the correct result!
Make sure you run this lines where the you only scan one “device” (local mount, nfs mount etc…) at a time!

  •  Find the largest file amount in current directory (it will only show the current level of directories)
for x in .*/ */; do if [ "$x" != './' ] && [ "$x" != '../' ]; then x=${x#*\'}; x=${x%\'*}; y=`find "$x"|wc -l`; echo -e "$y\t $x";fi; done|sort -k1 -h
  •  Find big directory under your current location (You will need to keep following the path deeper until you find the problematic directory)
du --block-size=1G --max-depth=2|sort -rh|head
  • Find large files:
    (Replace <path/to/dir/> with the path to the directory you want to search in)
find </path/to/dir/> -printf '%s %p\n'| sort -nr | head -10
  • Check your current location device free space (this work even if there is another mount under it):
df -h .


Entire Wiki in your EBook Reader

From time to time I find myself in a need to learn some new programming language or some new program/fs/feature. Now this thing is straight forward mostly – Find a good book on the subject and sit your ass down and read all about it. But alas! Some of the program I need to work with does not have a book on them, Moreover there are some good free guides and wikis out there for programming languages and programs.
The problem with these sites is that you will need an internet connection and a browser

Some time back I’ve needed to learn about LVM and BTRFS over a weekend and found some good guides for that. Yet the guides where online and I didn’t want my tablet or laptop with for the weekend while travelling.
I thought to myself – I have a small kindle with me and it’s been faithful these past years in reading all kinds of books – What about reading Wiki/Guide with my kindle? Continue reading Entire Wiki in your EBook Reader

Python textwrap – Fix Your Multiline Strings Indentation

I’ve been writing a lot in python the last couple of months and most of my code is for a CLI oriented environment (Mostly Linux/Unix shell).

One of the most important things when you create CLI is the help and usage string – Mostly because other people will try to use it – But lets face it, that script you’ve written a month ago? You’ve probably completely forgotten how it works.

So writing help or usage usually means that you work with an external file or multiline string. Continue reading Python textwrap – Fix Your Multiline Strings Indentation

SshFS for Windows and Why

As an IT and Information Security administrator in the past and the present I’ve needed time and time again to find elaborate solutions to protect the company I’m in (several companies in the past) against threats from outside and from within (also against user’s mistakes or stupidity).

One of the main issues IT guys face are user permissions and specifically data access permissions.
For example, you don’t want a user to be able to see company bills, employee’s salaries and management documents.

One particular issue that has risen again and again in several companies I’ve worked for is the main storage handled with NFS. Continue reading SshFS for Windows and Why

32Bit in 64Bit world – or how to run Fortinet SSLVPN Client on 64Bit Ubuntu + Debian

In this new and limitless world we have 64bit cpu’s and OS (Even MinuetOS is in 64bit)
Some legacy programs insist on running in 32bit because well … the company don’t have the resources, legacy reasons or any other reason why not to (I have no real idea actually why).
Most of the new OS can also run 32bit in 64bit OS, for example: windows 64bit can seamlessly use 32bit programs without hindering. Linux on the same hand – can do that too.

So this makes me mad that despite that 64bit OSs and software; not only available for the major part of the last 7-8 years; but also in the last 4 years 64bit OSs and softwares are the majority (by far) in the industry, and yet some companies think it is O.K. to deliver only 32bit software.

Last week a colleague came to me and asked me about his 64bit Ubuntu 13.10 – for some reason the Forticlient SSLVPN Client just won’t start working. The result is unpleasant after trying to run the file, the error:

bash: ./forticlientsslvpn: No such file or directory

A little search in the internet explained that this is 32bit software and you will need to install 32bit liberaries.
My first response was – why? I’ll just login to the Fortinet site and download the 64bit version right? Wrong – they didn’t have any (but oddly for window they had 32 and 64 bit)
So ok I’ve tried to install the liberaries all the answers in the web said:

$ sudo apt-get install ia32-libs
E: Unable to locate package ia32-libs

Wait? What? Why there is no package?
Rampaging through the forums and finally after 10 minutes I’ve got to a post saying – “Sorry folks – no more ia32-libs packages in Ubuntu 13.10 they removed it”
Wait what? What should I do now?
After alot oaf searching and a lucky guess I’ve decided to install gtklib 32bit – I recalled this is the one the Fortigate SSLVPN Client is using:

$ sudo apt-get install libgtk2.0-0:i386

That did the trick!
After the installation finished no more “No such file or directory” Error; now a new and more constructive error arise:

can't find

Well that’s easy – little search and:

$ sudo apt-cache search libSM
libsm6 Description: ...
$ subo apt-get install libsm6:i386

I’ve continued  on until I’ve installed them all.
Finally after 5 more minutes the Fortinet SSLVPN Client was running!

The trick was easy:
After an error for library arose I’ve search the package containing it either with “apt-cache” “apt-file” or Googled it.
After I’ve found which liberary I need to install I’ll install it with the suffix of “:386” (so: <PACKAGE_NAME>:386) this the 32bit libs will be installed.

This trick will probably work for any 32bit software.

For the impatient one here is what you need to run in order to get Fortinet SSLVPN Client running is:

$ sudo apt-get install libgtk2.0-0:i386 libsm6:i386 libstdc++6:i386
Ubuntu 16.04 updates:

By Arnon (thank you very much!):

sudo apt-get install libcanberra-gtk-module:i386

It appears that ubuntu 16.04 have spread the files over more packages now you should add:

For those who use Debian you can follow this instruction:

By Debian – (Thank you very much!)

Under Debian you need to do this first:

# dpkg –add-architecture i386
# apt-get update