The Injection

OK now nothing much really happened today in the office. I worked into the office by 7:45 am this morning, and had to wait till around 8:45 am till the Generator was turned on.

As soon as the gen came on, i went to my office, turned on the Server and did a system restore, to see if i could solve some of the problems we were having with our link. After the restore, i rebooted and things changed slightly. I ran a malware and a virus scan again, and found three malwares on the system. I removed them and then repaired kerio firewall. Over time i have come to find out that the particular malware we always contacted, had a way of messing up with the Kerio WinrouteFirewall. Everytime you removed it, you had to repair kerio WinrouteFirewall, because after removal of the malware, the system automaticall enables Internet Connection Sharing. Kerio WinrouteFirewall cannot run well when this is enabled becasue it does the internet sharing itself, so there will be a conflich of protocols.

Well after i cleaned up, reinstalled and restarted, things seemd to be much better, now we had just occasional increase in the response time. I have finally made up my mind to refomat the Server, as soon as we get our Kerio License. This decission i made because i think Kerio has injected a lot of things into the system, since its open and we have been running it without an antivirus and without the full funtionality of the Firewall.

We could not cross the Bridge today, because there was a heavy down poor, and you should know that riggers do not climb masts when the weather is not clear. The plan we had today was to bring down the two radios and then test them on the floor to see if they can connect ot each other at close range. If this is possible, then we know its either an allignment problem, riggers call this panning, or because of the amount of water that was found inside the radios, and inside the ethernet cable, the radios have lost there strenght and cannot transmit long range. If we have an issue with the signal strenght, then we might have to use a booster, if this is not the case, then we have to change the radios.

Like i said, today was a little ont the chill side, but i still ran up and down a couple of miles.

Ill keep you posted on what happens with the radios.

Tschuss.

Thel License

Tuesday and Friday had nothing really significant to the working experience, just that i decided not to stress myself unnecessarily. Our link was slow again in the office. We were having a response time from our ISP of over 3000 ms. After dong some troubleshooting here and there, i found out that when we had Kerio Firewall running, the response time
sky rocketed, but once you stop Kerio from running, everything became normal. I concluded, that this had to do with the expired License. We were running an expired License that was bought in a former partners name, apparently the other company had purchased a new License. Also, i knew that the Server had to be repaired or even reformatted and setup
right from scratch like a new baby.

Wednesday, through Friday made plans on getting a new Kerio Firewall and MailServer license.
Nothing really spectacular happened in the office, but even at that i never really had up to an hours time to sit down and have a quite time with the internet and a cup of coffee. There was always this thing to help someone do here or there. A lot of these are not in my Job
description, but i think everybody in the office thinks i know everything about computers, and that its my job to tell them where an icon is in MS Word, or how to carry out a particular task with it. I even get calls from the engineers in the office asking me to show them something in Primavera and Auto CAD. I let the engineers know that I'm not an engineer, and don't have a need to learn how to use those applications. For the other users who complain about something in MS Office, if its something they are supposed to know, i just let them know that they were supposed to be computer literate before getting the job. I'm not the person that is supposed to teach them basic computer appreciation. Some times they can be really annoying, but the job is
really fun.

Well that was my week, and i hope this week will be more interesting and we will be able to atleast pay for the License so that in a couple of days time we will get back to normal

The Bridge

The past week was sort of stressful, but i decided to take it easy and not to bother myself too much, because i knew the cause of our problems and we were going to take care of it.

Well Monday was interesting because. My MD complained that he could not work from his home. So i had to go over to check the problem. We have a bridged network from the office to the MD's house. Bridged networks are very much in use today, because its more cost effective than having a VSAT installation in all your office site. An organistion with a lot of branches in a state for example could use a Bridged network within that state. All they would need is the VSAT installation in there main main branch, then a radio link will be set up between the main office and the various locations. This means that every other location has to have a radio installed too. The organistation could now set up the wireless link between its main office and branch offices as a bridge, so that only radios in there network can associate with the access point in the main branch.

Now befor i left for my MD's place i called a Rigger. We got there and i started trouble shooting from inside the house. The radios we used in this wireless link are powered by POE's (Power Over Ethernet) adaptors. The POE, is a small device that powers wireless radios thorough the 4 unused wires in an etherenet cable. One end of the POE is connected to the lan, the other end goes straight to the wireless device. The POE is then powered with an adaptor. The advantage of POE is that one Ethernet cable connecting from the POE to a wireless radio can be used for both power and data. This removes the need for having a seperate power source for the radio.

After pinging the wireless radio a couple of times from the LAN inside my MD's home, i found out that the radio always freezes after a couple of minutes. When you restart the radio, by disconnecting it from the POE and conecting it back, i was able to ping the wireless radio in the house, and ping across to the radio in the office. After a couple of minutes, the link from the house to the office will start timeing out. When i first examined the POE, i found out that it was wet and there was water driping from the cable connecting the wirless radio on the mast. disconnecting the cable, i noticed that the pins that carried power to the radio were burnt. I just hope that this would not have caused a mojor damage to the radio.

I had to cut the cable between the POE and the radio short because the amount of water we found in the cable was not acceptable. We have a 40 metre ethernet cable running from the POE in the house to the wireless radio on the mast. It was suprising how water got into the cable and all the way to the POE. After recrimping the cable and running a couple of pings again, we decided that the radio on the mast had to be brought down, and the ethernet cable between the radio and the POE had to be changed.

The Rigger could not get acces to the radio, since the mast which it was on was being shared by my MD and the company next to him, and the security guard there did not let us in because his boss was not around to give him permission. We left MD's house and went back to the office to make sure everything was alright from that end. On getting to the office, the rigger climed up the mast, and on bringing the radio down, we found out that water some way found its way into the radio. During installation of the wireless radio, we never thought this would happen, because we thought these were outdoor equipments, and that they were made to resist to such conditions. Well maybe they were not made to withstand the kind of rain we experience in Nigeria.

After cleaning and sealing all teh edges of the radio, the rigger climed back and installed the radio on the office mast. We pinged the radio Wireless radio in the office, and everything seemd alright. We tried pinging across to the radio in MD's house, but to no avail. We scanned for other radios in the area, and we found a whole lot of other radios. So we concluded that from this end everything was ok and that the problem now must be from MD's side, since we did not have permission to bring down his radio. We were about leaving for his house when rain started falling. You know riggers cant work under that condition. We postponed the job till another day.

Spyware! Malware!! Virus!!!

This is coming late, because the weekend was very tough in the office.

I came into the office early on Friday, everything was going on fine until around 9:30 am when the link just became suddenly slow. I noticed this because all of a sudden I was not able to open web pages. The first thing I did was to ping the IP address of my ISP, to make sure that the link from there side was OK. I found out that I was loosing a lot of packets from there end. The next thing I did was ping my default gateway from inside the LAN and I still got the same loss of packets. I guessed a computer on the LAN must have been sending out a lot of broadcast, that would cause congestion on the LAN. On the other hand this could be a Virus, Spyware or Malware problem too. Just to make sure everything on the LAN was OK, I switched off all the Switches on the LAN to disconnect all the systems on the Network, then I pinged my default gateway of my LAN to see the response and I still got the same loss of packets.

When pinging your LAN's default gateway, you should get nothing higher than a 2ms response time. I was getting up to 4000ms. You could see how serious this situation was. I concluded that since the LAN was shut down and I was still getting this high response time, then the problem must be on the Server. I ran a Spyware scan on the Server, and it found two potentially dangerous threats. I tried removing them, but every time I removed them, they reinstalled themselves. This was serious. I could not even run Task Manager. The other option I had was to edit the registry and try and locate the registry key of the Spyware. I found the keys and then removed them. I also ran a boot time scan with Avast Antivirus on the system. A boot time scan is always good because the Antivirus software can scan all files and folders before the system starts up any processes. This is better because nothing would have been loaded so not even viruses or Spywares would have started before the scan. The Antivirus scan found two infections and removed them. The Server booted and I logged back in.

I scanned the system again for spywares and malware and nothing was found. That had been taken care of. I pinged the default gateway of the LAN and I was still getting high response time and loss of packets. I checked our firewall and everything seemed OK. I decided it was a physical layer problem. I pinged the loopback (127.0.0.1) address of the NIC (Network Interface Card) connected to the LAN, it responded. I also pinged the NIC's IP address which is the default gateway of the LAN and it also responded. So this meant the TCIP stack on the card was OK. I still decided to change the NIC card and see what another card does. So I had to look for a system that had an extra NIC card that was compatible with the HP Proliant Server. I was able to get one and I swapped it. I pinged again and things seemed better.

High Response Time

With that taken care of, I put on my
laptop and connected to our LAN, and tried browsing, but this was not
possible. I ping our default gateway, but I didn't get any response.
Now this was strange, because everything seemed to be perfect. I
checked our firewall next and found out that Kerio WinrouteFirewall
ties itself to the MAC address of the NIC it was installed with and
if you remember, I had already changed the NIC of the LAN interface.
So I tried repairing the FireWall to see if this will solve the
issue, but it didn't work. Then I completely uninstalled the Firewall
and reinstalled it. I then reconfigured the firewall with the
configuration wizard and restarted the Server. Now I was sure that
everything would work normally again, but there were still some
issues. So the last steps I took was to copy the configuration files
of the firewall we had backed up a couple of months ago back into the
installation folder of the firewall and restarted it. After this
step, everything seemed to work normally again.

The only problem we
had with this now is that the config files I copied had some ip
address filtering on it. This filtering enabled only a particular set
of users to be able to access the Internet. With this filtering, HTTP
access was disabled for some particular users and only the management
group, or ip addresses of the mangers were allowed HTTP access. All
other users only had access to POP 3, IMAP and SMTP. These Protocols
allowed only access to email. These security measures were
implemented so unauthorized users would not access the Internet
during working ours. But management had changed there mind towards
this filtering. So the final thing I did on the Firewall was to
remove the filtering on it, and everything was back to normal. I
backed up the present configuration of the Firewall, so that I would
not have to go through this rigorous process to get things back to
normal.

This was all very strenuous and tasking
on me, since I had to stand on my feet almost through out the whole
day, to make sure things came back to normal. I also had to come to
the office on Saturday, to make sure things were in order. I had to
make sure everything was normal on Monday, because everything in the
office like a lot of other businesses today depend on the Internet to
function.

Well like every other day, Friday was
fun, but like I said stressful. A network administrator always has to
expect such days, because they test you and the end always make you
better and more experienced.

Poping

Welcome back. Today was hectic.

After solving some issues yesterday, I came to the office this morning and what awaits me? Three users made complaints of not being able to access there emails. As a network Administrator, one of your core skills an competence is that you have to be patient and also be a good listener, because some of the complaints I always get from users are very annoying. You also have to understand that in an organization, because there is what you call division of labour no one wants to do what they think is another man's job.

One of the users complain was easily solved because all I had to do was run up to her office, checked her desktop and I just pin pointed the problem. She could not access her mails because she was not connected to the network. Apparently, when the cleaners where doing there jobs this morning, they mistakenly unplugged her cable from the face plate. The cleaners are very good at interfering with my job. I run a couple of kilometers weekly through our office just to plug a users Ethernet cable back into the network.

The second users problem now was that she could pop her mails from the Mail Server, but she was popping old mails from 2006. What! This Is 2008 how did that come about? Well I figured this was an effect of the small changes we made yesterday to our MailServer. Remember we uninstalled our mail server completely yesterday. The version that was uninstalled was KMS 6.4. Now because our subscription license expired sometime in November 2007, and we did not readily have the 6.4 version ready for installation, I decided to install Kerio MailServer 6.3. Kerio MailServer runs forever without a subscription license. The only problem is that you will not be able to upgrade to latest versions and there will be no Virus Definition Update for the inbuilt McAfee Antivirus. That said, I also found out that because of the changes we made yesterday, users were not able to log into there Webmail interface of the MailServer. This was really strange because I never thought downgrading would cause this kind of a problem. I checked on our MailServer to see how much mails this lady had in her Inbox storage, and I found out that she only had about 3mb of mails and I was sure that she had a lot of junk mails that would occupy some 1Mb. So I just decided to delete her mail account, which will delete her mail storage folder. Then I created it again.

Poping Continued

The complaint of the last user was more confusing. She could not log POP her mails because she was not being authenticated on the mail server. This was strange because this was exactly the problem we had we had with the whole system yesterday. Users were being asked to input there user names and passwords, and these were already configured on there MS Outlook client and on the MailServer. I figured out that she had her mail client open yesterday while I was messing around with the mail server, and this must have cause an authentication problem on her Mail client. I thought of a solution to this but it didn't come immediately. So I just decided to chill for a while. Later in the day after lunch, I just went over to her system, excused her,
1.Opened Outlook, removed the mail account that was configured on it and created another one.
2.I came back to the server room, opened the MailServer configured the same account I just created on her system in step one above on the Server and applied the settings.
3.Then I selected her old account to remove it. In the process of removing it, I made sure I checked the box that said copy mail folder to another account, I selected the new account I just created in step 2 above as the account that should be copied to. Then her previous account was deleted from the MailServer.
4.I went over to her system again and POPed with the new account that was created in step one above, it didn't give me the authentication problem again. I now knew that my problem was 75% solved.
5.Next I went back to my office ie the Server room, I recreated her old account, thats the account she had before Step 1 above. Take note that this account will now be a fresh account on the MailServer, i applied the settings,
6.Then I stopped the MailServer from running. I then copied her old Mail folder that was backed up in step 3 above from that account to the mail folder of the account I created in step 5.
7.I restarted the MailServer, went back to her system, removed the account I created in Step 1 and reconfigured the account she had before Step 1. I POPed and WOW!!! everything was working as normal again.

Please take note that whenever you are working with a client server application, there has to be no link between the client side and the Server side when you are doing any configuration on the Server side. This causes a lot of problems 90% of the times.

Network Administrators have to be very patient, good listeners, and one major thing is you need to know how to think logically so you can easily figure out the cause of problems. A lot of times when you come across some problems, you never understand how they could have happened int first place. From what I have noticed in my line of work, anything that is engineered must come up with a problem some time.

Like I said today was hectic, but not as hectic as yesterday. I wonder what tomorrow brings. Fridays are normally very quite.

WELCOME

Hello, and welcome to my Blog. All ill be talking about in this Blog will be my daily activities as a Network Administrator. The blog will be educative, since it will take you into the daily life of a Network Administrator. You will face challenges with me and also be victorious with me at the end of each day. I hope we will learn from each other from my daily postings. Please do feel free to drop a comment if i have blessed you, or helped solve a problem. Also if you have any suggestions ill be glad to hear.

Thank you.

Corrupt Configuration File

Today was hectic. I was at the location of the HSE training organized by Exterran an Oil and Gas company. Barely 30 minutes of my arrival, I received a call from my GM saying that he was not able to access his mail in the office. Now this came as a shock, because as at yesterday before I left the office, everything was just perfect, I also thought that my colleague will be in the office. I gave my colleague a call and guess what? He was not felling fine and said he will not make it to the office till about afternoon time. So I had to make my way back to the office with the Mopol escort and a 504 wagon that needs to be scraped. I got to the office around 10:30 am and to my surprise the whole office could not pop or send there mails, because the mail server was acting up. We make use of Kerio MailServer and KerioWinrout Firewall in the office.

Well what do I do? The first thing you start doing is try to get to understand the problem. I found out that users were not being allowed access into the mail server including the administrator, thats me. It was annoying. The funny thing with this problem was that the mail server was running on the Server, but because of the corrupt configuration file, the mail server was not allowing connection to it both from inside the Intranet and from the Internet. In other words, mails from outside our domain will be bounced back with a delivery report that the mail is undeliverable, becaue the mail server is not responding or does not exist.

So I had to tackle this problem from the bottom up. First I backed up all the configuration and user files from the mail server directory to a Network drive. Then next I tried repairing the mail server but this didn't work. What next? I uninstalled the mail server software completely, restarted the machine and then reinstalled it again. Now this seemed to have solved the problem, because it now contained a new configuration file and an empty user configuratin file. This posed another problem because the the Mail Server had to be reconfigured from scratch again plus i still had to create all user profiles and email addresses for the organization and I did not have that much time to play with, because users that needed there mails to get there jobs done were on my neck.

OK now the next step I took was to stop the Mail Server, then copied the configuration and user files from the external drive back to the installation directory of the Mail Server. Then I started the Mail Server and found out that the same problem was back again. It was then I knew that the configuration file was corrupt. The last step I took was to replace the present configuration file with one that I had backed up some months back. I restarted the Server and everything was back to normal again. Only complaints were that all mails that were sent from this morning till the time I resolved the problem had to be resent. Not a serious issue.

This was really an hectic day, but I came out victorious at the end of it. Network administrators are like doctors. They can be called upon at odd ours of the day depending on the organization you work for. So always be ready. And do not rush to find a quick way out of a problem, except you are doing it temporarily. Always try as much as possible to do the right thing, so you wont have to solve the same problem in the nearest future.

Ill give you an update of what my tomorrow looks like