SCOE Outage

I received the following notice from SCOE:

We are planning an upgrade to our firewall on Sunday September 25.
This will cause intermittent Internet outages on Sunday while we work.

We expect to have everything operating normally before Monday morning.

A SCOE outage will equate to a loss of internet access on our network.  Things like web browsing, e-mail, PowerSchool, VPN access, etc. will not be available during the SCOE outage.

Dunbar Phone Outage

On the afternoon of Sept 16, 2011, an IT support request was received from Dunbar School for their primary phone line, 707-935-6070.  The IT Department investigated the issue and confirmed that there was no tone at the AT&T interface.  AT&T CMAC was contacted and their test confirmed line problems in the AT&T network.  AT&T will be on site Monday morning to work on the problem.  In the meantime, all phone calls to Dunbar are being routed by AT&T to an old voicemail.

UPDATE 9/19/2011 09:45 AM:
Phone service was restored to Dunbar on Saturday morning by AT&T.

E-Mail Delivery Issues

At about 9:00 this morning I began to receive word that e-mail messages from district accounts were being rejected by Google/Gmail and a few other sites.  I investigated these reports and confirmed that messages were being rejected.  The problem that Google reported was that the IP address of our outgoing mail servers were not resolvable externally.  This is a standard check to keep spam from flooding their system.  I contacted SCOE and they added external DNS entries so Google and other sites could perform reverse lookups on our SMTP servers' IP addresses.  After the entries were created, I sent test messages to Gmail and confirmed delivery.

This behavior was caused by a Routing Connector that was created by Exchange 2010 when a Hub Transport server was introduced into our Exchange organization.  Exchange 2010 forced our Exchange 2003 system to route all outgoing mail through one of the new Exchange Hub Transport servers that was not prepared for deployment.  Luckily it was not difficult to identify or fix.

PowerSchool State Reporting Update 2011-13

California Update 2011-13 was applied to the PowerSchool Server. The update required a restart of the PowerSchool Application at 4:54 PM on Tuesday, July 11, 2011.

PowerSchool Service Interruption

On Tuesday, June 14, 2011 from 7:45 AM until 8:45 AM PowerSchool was unavailable.  The application encountered an error and hung while it was shutting down.  When the application was brought back up, a security setting had been changed to allow only 2 concurrent logins.  This setting was corrected, the PowerSchool application was restarted, and service was restored.

SVHS Network and Connectivity Issue

From 12:30 PM until 2:00 PM on Tuesday, May 31, 2011, the SVHS network was experiencing connectivity issues.  PowerSchool, e-mail and web access were affected.  The cause was determined to be a broadcast storm that was created by an improperly configured network adapter on a server.  The problem was corrected and service was returned to normal.

PowerSchool State Reporting Update 2011-10

California Update 2011-10 was applied to the PowerSchool Server.  The update required a restart of the PowerSchool Application at 10:54 AM on Tuesday, May 24, 2011.

SVHS Wireless Unscheduled Maintenance

On May 16, 2011 the IT Department made unscheduled configuration changes to the "SVHS Staff" wireless network at Sonoma Valley High School.  From 11:30 AM - 12:00 PM the wireless network was not available. 

The configuration changes were made to address an issue that was causing the admin network to be unavailable to some users.  The admin network was over subscribed due to a large increase in the number of wireless users accessing the network.  It was discovered that the increase was a result of the wireless password had compromised.  Some students found out what the password was and were distributing it to their peers.  A multitude of devices including iPhones and iPods were found connected to the wireless network.  When this was discovered, the network was reconfigured and regular service was restored at noon. 

As a result of the compromised password, the password for the "SVHS Staff" wireless network will be changed Friday, May 20, 2011 at 5:00 PM.  SVHS Staff members will need to contact the office or their tech coach to acquire the new password.

ShoreTel Software Update

On May 16, 2011 the IT Department upgraded our ShoreTel phone system to version 10.2 Build 15.42.8562.  The upgrade was to address a problem with exporting users from the web console.  The phone system was down at 4:30 PM for about 15 minutes to complete the update.

PowerSchool Application Restart

At 11:05 AM on April 11, 2011,  the PowerSchool application was restarted because it had become sluggish and unresponsive. 

The server console revealed that the application had a runtime error screen.  Unfortunately, the application was unresponsive so the popup could not be read.  A review of the logs will be necessary to determine the cause of the error.

Webmail Outage

From Saturday, April 9, 2011 at 7:45 PM until 2:00 PM Sunday, April 10, 2011 access to the e-mail server was interrupted.  No mail was lost as all e-mail that was delivered to the district during that time was held by our second e-mail server until access was restored.
 
This outage was caused by an error by the network card on the server.  It's TCP/IP stack was still functioning, but was unable to ping it's gateway or anything on the network segement.  Server was rebooted and service returned.

The Blackberry server was also affected and services needed to be restarted.