The Email Migration
-
About 9 months ago the subject of student email usage came up at a meeting. The aging Exchange 2003 servers at the campuses were filling up and breaking down, students were limited to a paltry 50 meg, and attachments were limited to 2 meg. This wasn't new news to the campus LAN support personnel - they had been begging and pleading for help from Cental IT for some time. Eventually another meeting was called, one that included Bob, the tech ops director, and the CIO, a women who had just recently took over the job. She had come in guns-ablazing and had already been not-so-secretly remprimanded on several occasions by the assistant dean for student services for overstepping her bounds and basically being a bitch to everyone.
But I digress.
The campus LAN support personnel decided that the best option would be to spend about $25,000 to upgrade the 8 campus servers. New hardware would be purchased. Licensing for Exchange 2007 was included in the existing campus select agreement so that wouldn't be an issue. With new servers and larger hard drives both faculty and students could be satisfied.
The CIO and Bob decided instead that, rather than spend money to replace the servers, they would "invest in the cloud". "The cloud" was the way to go, we were told. Several options were examined personally by the CIO and Bob and Cloud.edu was chosen as the preferred service provider.
Cloud.edu
At first the campus LAN support personnel thought it would be a great idea. It'd be low to no cost for the campuses, students could have much larger storage, send larger attachments, retrieve emails on their mobile devices and we wouldn't have to support Exchange for 25,000 students anymore.
As some other details emerged however, caution began to set in. The Cloud.edu solution didn't come with an SLA. The campus LAN support personnel were still going to be the primary support providers. If something broke and neither the campus LAN support personnel nor Cloud.edu's forum support could fix it, that was it - the student would just be without email. Lastly, the email migration was slated for the week before final exams started AND at the end of a major Active Directory migration. Deans complained and were summarily ignored by the CIO.The CIO kept sending out updates, ignoring pleas to reconsider the move for after finals week during winter break when email availability for students would be far less crucial. The CIO insisted that there would be no problems. Evan, the consultant & dedicated project manager, agreed.
The Consultant
The consultant worked for a company that even Google can't find. The only information provided about him to the campus LAN support personnel was that the CIO had worked with him before on a similar project and that he had had the lowest bid. His previous experience, technical, project management, or otherwise, was not defined anywhere.
Evan decided to rule over the project as project manager with an iron fist. "We WILL work as a team!" he said sharply, glaring at each of us like a drill sergeant would a first-day recruit. "We WILL be successful!" We WILL communicate consistently! And we WILL use SharePoint to do so!"
The CIO had recruited several campus LAN support personnel to assist with migration tasks, such as scripting and on-site support. During the project Evan decided that some of the campus LAN support personnel who had volunteered were not good enough and summarily removed them from the project. Later, he fired one of his own men and replaced him with a ghost named Bill. No one knows why he was fired but by this point the campus LAN support personnel were already accustomed to seeing the "Project resource change notification" emails that Evan liked to send out to remind everyone that he was in charge.
If Evan was the brains of the operation Bill was the gears. While Evan would blast emails out left and right at the beginning, Bill could be heard sighing once in a while in the background on a teleconference. When things broke during the testing phase Bill and Evan disappeared for a week. By the time they came back on email the script had been modified slightly and was limping along somewhat. Evan declared that this would be good enough. "We MUST persist!"
The Migration
The migration started over a weekend the week before finals. Students were given new instructions on how to set their passwords, how to login to Cloud.edu, and how to move emails from their old mailbox to the new one. For that to happen, the students had to be made aware that emails would only arrive at the new mailbox location and not at the old email location that was still hosted at the campus Exchange servers. In order to save vital emails students had a week to forward those emails they wanted to save to themselves.
The campus LAN support personnel were quickly trained on support. The new webpages were up. It was go time. On Monday the campus LAN support personnel held their collective breaths.
An issue or two popped up but was quickly tracked down to a password synch issue. The campus LAN support personnel breathed out a bit. Another issue turned out to be a password that was over 16 characters long not working properly. The student was counseled against using such a strong password by Central IT. Breaths came a little easier.
By noon everything seemed to be working fine. A few training issues popped up here and there but nothing that couldn't be quickly fixed with a new password or a document that was linked on the web page.
The the first bounce message came. Cloud.edu wasn't accepting emails, returning a 550 - IP blacklisted for phishing behavior. More and more bounces arrived. An issue was opened with Central IT. The cause was discovered after a few hours: Apparently no one told Cloud.edu that thousands of students would be forwarding emails from their old boxes to the new ones so the 4 IP addresses given to them during the setup phase became quickly blacklisted for phishing activity.
The matter was resolved after the misunderstanding was cleared up. Bill and Evan were silent during the entire process.
-
Nice story, It's unusual for such a long post in the side bar to be so readable...
But, err, yea, I guess TRWTFTM is that they didn't just migrate to google apps which is free for educational institutes and very liitle hastle to set up...
-
Hey, sounds like what happened to my alma mater, except it was a migration from GroupWise to Exchange, and from there to Live Office or whatever the heck MS's abomination "login through Hotmail!" interface is called. I've since abandoned my "lifetime" @foo.edu address which I can no longer access for my @cs.foo.edu which runs Postfix on a machine I can actually SSH into.
Would this institution happen to be located somewhere in the state of Maryland?
-
No, but it sounds like you had a similar scenario. Cloud.edu may be the service you've named :)
-
@citking said:
Lastly, the email migration was slated for the week before final exams started AND at the end of a major Active Directory migration. Deans complained and were summarily ignored by the CIO.The CIO kept sending out updates, ignoring pleas to reconsider the move for after finals week during winter break when email availability for students would be far less crucial.
While the choice of the cloud.edu solution seems potentially boneheaded, this is outright stupid. Sounds to me like the CIO had vacation plans that she didn't want to break.
Also: Is it common these days to run Exchange servers at a university? What happened to good old Unix email?
-
@cconroy said:
Also: Is it common these days to run Exchange servers at a university? What happened to good old Unix email?
Color monitors were invented.
-
TRWTF is that it wasn't a bigger disaster.
-
@Heron said:
TRWTF is that it wasn't a bigger disaster.
Agreed. Considering the incompence and poor planning, this should have by all rights been a total trainwreck. Instead this is more like a minivanwreck.
-
@cconroy said:
Is it common these days to run Exchange servers at a university? What happened to good old Unix email?
My Uni ran Exchange, and has recently migrated to the Windows Live something or other that they host in the magical cloud. They're migrating students slowly and I haven't been reached yet. Hopefully this means that they planned ahead and realised that if they hit problems then they want it to be for as few students as possible. I just set up a dedicated gmail account to retrieve my email anyway.
-
My school stopped providing @wtfu.edu addresses several years ago; they just tell students to go use gmail or hotmail or whatever. (They may provide @wtfu.edu forwards, I don't recall.)
The CS department still provides @cs.wtfu.edu addresses to CS students, for those CS students who are scared of free webmail.
