Daniel Eran Dilger in San Francisco
Random header image... Refresh for more!

Microsoft’s Pink/Danger backup problem blamed on Roz Ho

Roz Ho

Daniel Eran Dilger

A fourth source has added new insight into Microsoft’s Danger datacenter disaster, pinning the root cause directly on Pink Project leader Roz Ho rather than Microsoft’s third party vendors, which the company is itself seeking to scapegoat.
.

Existing reports have all described Microsoft’s Danger datacenter problem as resulting from a maintenance event related to a failed SAN firmware update. The story is that the SAN update failed and Microsoft was subsequently unable to restore the data from backup. This leaves a lot unanswered; fortunately, the new source fills in those gaps.

Microsoft. Your potential problem: our third party vendors’ lack of passion.

Meanwhile, according to a new report by Mary Jo Foley, Microsoft has been (understandably) scrambling to distance itself from the debacle, publicly stating that the Danger datacenter operations were built using a Sun SAN solution and running an Oracle database. (A Danger source earlier casually described the SAN upgrade as involving EMC; the company was quick to insist that it was not involved, even though the SAN’s vendor, as you’ll soon see, was not really the reason for Microsoft’s datacenter failure at all.)

Foley’s article also reported having heard that it was Hitachi Data Systems that performed the update for Microsoft acting as its contractor, apparently because nobody at Microsoft knows how to eat dogfood that hasn’t churned out of the Visual Studio grinder. Well, that and the fact that Microsoft doesn’t sell SAN firmware, which some pedantic critics latched onto to suggest that my source’s use of the term “dogfooding” was such a problem for them that they couldn’t finish considering the rest of the article.

To clarify, my source used “dogfooding” in the context of ridiculing the phrase, specifically writing: “keep in mind that Microsoft has a real hard-on for running the latest and greatest ‘dogfood’ (God, I loathe that word, especially the ‘dogfooding’ verb form, and I refused to use it without sarcasm in my tenure there).”

Apart from casually referencing EMC, the source also noted that Danger’s data center ran using a backend built on Linux, Sun hardware and Oracle RAC, as I reported. So please, united band of twittering pendants, keep spinning your fan blades in your vacuum of rage if that’s the best you got.

Exclusive: Pink Danger leaks from Microsoft’s Windows Phone
Microsoft’s Danger SideKick data loss casts dark on cloud computing

They also laughed derisively at the suggestion of sabotage.

Foley also wrote, “I’ve also heard that foul play has not been ruled out because the failure was so catastrophic and seemingly deliberate. Microsoft is supposedly continuing to do a full investigation.” She had earlier linked to my article outlining insider reports that indicated the issue might be related to either overly ambitious upgrades or intentional damage set up by a disgruntled employee.

On an unrelated side note: Foley is a good sport considering that a couple years ago I mercilessly lampooned her for writing that Apple had “licensed the Exchange ActiveSync licensing protocol” in order to gain compatibility with corporate email systems on the iPhone. Her article was singled out my criticism despite being merely one bucket in the torrent of naysaying ZDNet had unleashed in its efforts to drown Apple’s new phone in muddy FUD waters.

I should also note that while I was correct in pointing out at the time that “Apple would still have to write its own implementation of ActiveSync for the iPhone,” I also overstated my case in portraying Foley as Gollum, blindly focused on Windows as ‘the precious’ (an unflattering graphic which has subsequently become a top image hit in Google when you search for her name).

I’m glad she apparently didn’t take it personally, and I apologize if you did, Mary Jo. I also have to humbly admit that, under all the layers of Apple-ostracism that she is apparently contractually obligated to provide for ZDNet, Foley scooped the world on announcing that Apple was in fact licensing EAS for the iPhone, and did so almost a year before Apple publicly announced it. I didn’t directly contradict her scoop back then, but some readers who only took a cursory look at what I wrote were left with the idea that I had, so I might as well apologize for that too while I’m off on this wild Lord of the Rings tangent.

Microsoft’s Sidekick/Pink problems blamed on dogfooding and sabotage

Anyways… here’s what you came here to read

With Microsoft frantically trying to associate its Danger fiasco with everyone else possible in the industry, and particularly upon its direct competitors Sun and Oracle, it failed to point out that an awful lot of enterprise datacenters are running Sun and Oracle, and yet there aren’t regular outages that last for weeks and announce having lost all their users’ data. In fact, the losses Microsoft experienced (and its shoulder shrug response to T-Mobile’s million Sidekick users) are virtually unprecedented in the industry.

Brief service availability problems and isolated occurrences of data lost have hit Gmail and MobileMe, and generate big headlines each time. The difference is that data related to Gmail and MobileMe are designed to be backed up by the user; Danger’s Sidekick was designed to use a mission critical service operating under the assumption that users did not need to act to secure their data themselves. In fact, users had no capability to back up their devices’ data. Even the Outlook desktop sync tool (which Danger provided as an extra fee) synced with the cloud service, not the local device.

This means the Danger disaster is a lot closer to Microsoft’s previous fiascos in trying to migrate Sun-based services like HoTMaiL and WebTV to Windows than with the occasional service interruptions that impact other cloud services. RIM’s BlackBerry service goes off at regular intervals, but users don’t typically have their devices completely wiped when that happens.

The buck gets passed when the bucks stop.

Despite all of its attempts to pin blame elsewhere, Microsoft was running the Danger service not just as a convenience to users but under contractual obligation to T-Mobile.

This wasn’t just a case of Microsoft just deciding that it was tired of serving customers who weren’t paying it enough, like the times it has pulled the plug on its DRM authentication servers, stranding users of Yahoo Music, MSN Music, Major League Baseball, and everyone else who had invested in a Microsoft PlaysForSure partnership that the company decided it no longer needed to honor.

Microsoft had big money on the line with Danger: a major partner joined at the Hiptop, a million paying customers, an SLA contract to service, and top-tier enterprise grade equipment and software running the thing. The problem wasn’t Sun equipment or Oracle software, it was something unique to the way Microsoft was running its very serious obligation.

Microsoft’s Zune, Vista, and Windows Mobile 7 Strategy vs the iPhone (DRM)

It was Microsoft management.

According to the source, the real problem was that a Microsoft manager directed the technicians performing scheduled maintenance to work without a safety net in order to save time and money. The insider reported:

“In preparation for this [SAN] upgrade, they were performing a backup, but it was 2 days into a 6 day backup procedure (it’s a lot of data). Someone from Microsoft (Roz Ho) told them to stop the backup procedure and proceed with the upgrade after assurances from Hitachi that a backup wasn’t necessary. This was done against the objections of Danger engineers.

”Now, they had a backup from a couple of months ago, but they only had the SAN space for a single backup. Because they started a new backup, they had to remove the old one. If they hadn’t done a backup at all, they’d still have the previous backup to fall back on.

“Anyway, after the SAN upgrade, disks started ‘disappearing.’ Logically, Oracle [software] freaked out and started trying to recover, which just made the damage worse.”

The problem with this report is that is places the blame, not on a complex Oracle deployment, not on bad SAN hardware or a firmware glitch, not a disgruntled employee with inappropriate levels of access to a mission critical service, but squarely upon Microsoft management.

This management decision was (allegedly) made by the same group within Microsoft that authorized spending $500 million to acquire Danger and take on accountability for its SLA with T-Mobile, botched the development of Pink, spent three years and untold sums developing the Zune brand so that users could sit through TV-style ads before launching Chess on a handheld, lost billions on Xbox and set a new ‘low-water mark’ in consumer device reliability, boondoggled Windows Mobile to the point where even Gartner can’t say nice things about it, and which has responded to the criticism of Apple’s App Store by launching its own software store with far more rules, significant new fees, and far fewer desirable offerings.

This latest report does not exactly fail to fit in with the general incompetence that emanates from Microsoft’s Entertainment and Devices Division. Rather, it seems entirely credible given the increasingly toxic relationship that has been brewing between Microsoft’s reality-challenged managers and its often frustrated engineers.

Why Can’t Microsoft Develop Software for Zune HD?
Microsoft uses adware model to pay for Zune HD apps
Gartner declares Android a second place winner in 2012. Why?
Microsoft sells restrictive new WiMo Marketplace via iPhone ads
Lance Ulanoff and Robbie Bach explain why library size doesn’t matter

A miraculous resurrection of data

How does Microsoft back itself out of this crisis? How about denial. Foley also reported that Microsoft has now announced (but not yet delivered) a reversal of its earlier summation that all of Danger’s data was lost.

In a website message signed by Roz Ho, the company stated, “We are pleased to report that we have recovered most, if not all, customer data for those Sidekick customers whose data was affected by the recent outage. We plan to begin restoring users’ personal data as soon as possible, starting with personal contacts, after we have validated the data and our restoration plan. We will then continue to work around the clock to restore data to all affected users, including calendar, notes, tasks, photographs and high scores, as quickly as possible.”

If the company has stumbled upon a novel recovery avenue or some unknown backup that somehow remained missing for nearly two weeks, then this is great news for Sidekick users and helps to wipe some of the egg from the company’s cloud computing services face, although the situation still remains as the worst datacenter failure to ever impact mobile users as well as one of the most absurd responses pertaining to lost data as well.

The public notice Microsoft posted also states “we have made changes to improve the overall stability of the Sidekick service and initiated a more resilient backup process to ensure that the integrity of our database backups is maintained” (italics mine). Certainly, if the source is correct and Ho ordered the work to continue without a backup, this is an understatement to say the least.

Is this just vaPoRware?

However, Microsoft is also well known for advertising bullshit it can’t deliver. Bill Gates talked up OS/2, floated a vision of Cairo that never materialized, falsely proclaimed himself the Moses of tablet computing, and blew so much vaporware at competitors (Bob, ActiveMovie, DirectMovie, Surround Video, Chromeffects, WinFS, SPOT, Mira, PlaysForSure, Advanced Streaming Format, Soapbox, Longhorn, Surface, Natal, Courier) that it wouldn’t exactly be a surprise if the company decided that the best way to compete with bad news was to generate some distracting good news that just never seemed to materialize after people’s attention spans moved on.

If your attention span wanes after a few of years, it might have slipped your mind that Microsoft demonstrated a blockbuster new version of Windows back in 2003, over a half-decade ago. Tomorrow’s Windows 7 still pales in comparison to the vapor Microsoft issued at the time, as this PDC2003 video highlights. Special highlight: a Pink song that edits out lyrics about “kicking ass.”

Vaporware is exactly Microsoft’s core competency as a company. I hope I’m wrong, and the million Sidekick users who depended upon Microsoft get their data back. But this weasel-worded announcement, issued nearly two weeks after the initial problem, suggests the possibility that the company primarily hopes to provide a pat answer for Windows Enthusiasts to use when denying that there was ever problem.

Ho wrote, “We will work with T-Mobile to post the next update on data restoration timing no later than Saturday.” That’s a status update two weeks after the problem first appeared. And it’s not a recovery due date, just a progress update. Who, two weeks after losing their contacts and other data, isn’t going to just move on and scrounge together their information from other sources?

If Microsoft strings along users long enough, it will be able to pat itself on the back with a “mission accomplished” even if it ultimately never actually delivered anything. It’s like saying you’ll call somebody back after a date and then just waiting until they figure out that you’re not really interested. After two weeks, the party on the other end begins blaming itself for waiting around.

1990-1995: Microsoft’s Yellow Road to Cairo
CES: Fear and Loathing in Las Vegas

Is it real or is it Microsoft?

If Microsoft can deliver even most of most users’ data, it will be a relief to those users who relied upon it but it won’t erase the the reason why those users lost their data in the first place. The available evidence says this was not because of some unfortunate, unforeseen accident involving a very complex situation, but because Microsoft management decided to play fast and loose with users’ data just to save time and money. That’s pretty outrageous.

If instead it turns out that this latest announcement is just a public relations stunt designed to deflect criticism away from the company until observers decide that the personal pictures and contacts of a bunch of Sidekick-bearing kids wasn’t really anything that mattered too much anyway, then Microsoft’s management failure and all associated lessons pertaining to partnering with the company will simply be erased like so much Danger data. This is beyond outrageous.

And really, the fact that Microsoft is officially trying to associate its datacenter problems with Sun hardware or Oracle software is additional evidence that the company is irresponsible and disingenuous on a professional level. If the company has actually found a way to recover users’ data, it will mean that, despite all sorts of incompetence on Microsoft’s part, a solution built using competitive technology from Sun and Oracle is pretty damned resilient. But Microsoft won’t tell you that of course. It’s too busy suggesting the fearsome idea that products from competing vendors might introduce uncertainty and risk, the exact opposite of the truth.

Further, with this announcement, even if the company has no real data to recover, it will have erected a plausible story for denying anything significant ever happened. Know somebody who actually lost their important Sidekick data? You’ll be able to write them off as “one of the few who didn’t benefit from Microsoft’s miraculous data recovery.” It will be their word against Microsoft’s PR. Nobody will have records of who was impacted and whose data was recovered apart from Microsoft and probably T-Mobile, and the provider will likely have its records sealed by court order when it gets its big SLA settlement from Microsoft.

This all happened before

If this sounds like a conspiracy theory, take the day off and read through the archives of one of the monopoly trials against Microsoft. The company has regularly pursued a combination of criminal and incompetent activities that it was regularly able to later hide underneath big settlements, except for the occasions where court documents have been made public. They’re all quite damning.

There’s plenty of examples of Microsoft’s criminal past being sealed up behind secret settlement payouts:

  • A few years ago Microsoft settled with IBM in an antitrust suit involving OS/2 and IBM’s Lotus SmartSuite applications to the tune of $775 million.
  • Microsoft paid Novell $539 million to settle its antitrust suit over the NetWare operating system, and Microsoft is still being sued by Novell over claims related to WordPerfect.
  • Microsoft paid Palm over $23 million to settle an antitrust suit over the unfinished BeOS.
  • Microsoft settled with Sun in an agreement that included $700 million in antitrust and $900 million in patent infringements, both related to Java.
  • Microsoft paid AOL $750 million to settle the antitrust suit over Netscape.
  • And of course, during the 1997 return of Steve Jobs, Apple settled its San Francisco Canyon case which involved the outright theft of QuickTime code by Microsoft, as part of a secret settlement that involved a number of issues ranging from Office and Internet Explorer for Mac, to a cross licensing agreement, to a truce that prevented Apple from working on voice technologies.

The common thread that runs through the entire, multifaceted Pink/Danger imbroglio is that Microsoft’s management is criminally inept. But rather that being chastised to the point where real changes are made, it appears that the Microsoft-serving tech media is primarily concerned with moving past this issue so that the ineptitude can continue without causing any sort of unpleasant shakeup.

Which is exactly why dinosaurs die and empires collapse.

Mac Office, $150 Million, and the Story Nobody Covered
Readers Write About Mac Office, $150 Million

45 comments

1 lmasanti { 10.16.09 at 12:46 am }

I did not pay too much attention to the subject but I do remember that, somewhere, they say something about paying like $100.- to each user whose data was lost.
If true, this could be the “evidence” needed not to let winfanboy forget about the incidence.

2 The real story behind Microsoft/Sidekick dataloss? { 10.16.09 at 1:05 am }

[...] RoughlyDrafted provides some biting criticism and information from a source close to the situation. A fourth source has added new insight into Microsoft’s Danger datacenter disaster, pinning the root cause directly on Pink Project leader Roz Ho rather than Microsoft’s third party vendors, which the company is itself seeking to scapegoat. . Existing reports have all described Microsoft’s Danger datacenter problem as resulting from a maintenance event related to a failed SAN firmware update. The story is that the SAN update failed and Microsoft was subsequently unable to restore the data from backup. This leaves a lot unanswered; fortunately, the new source fills in those gaps. [...]

3 ronhip { 10.16.09 at 1:48 am }

Let’s not forget the wholesale theft of CP/M to create the original DOS…

4 Berend Schotanus { 10.16.09 at 3:32 am }

This is Shakespearian drama. When it comes to real power the experience over and over again is that conventional juridical or scientific documents wont do and the only form to describe it is by any form of art, be it a book, an opera, a movie. There are some beautiful examples out there.

“There’s plenty of examples of Microsoft’s criminal past being sealed up behind secret settlement payouts … $775 million … $539 million … $23 million … $700 million … $900 million … $750 million”

So the problem is that the truth is for sale. The noble juridical and scientific approaches are limited in that sense that they require one universal truth and they are only effective (and very much so) when there is a dominant power that can enforce this single truth.
The benefit of art is that it inherently assumes truth is a matter of perspective. The artist chooses to portray the criminal from his own perspective without necessarily denying the perspective of the criminal. The artist chooses his own goals and his perspective might not be for sale. This puts the criminal in a much weaker position because he can neither buy the artists perspective, nor present his own perspective any longer as the only truth.

“Which is exactly why dinosaurs die and empires collapse.”

I think this is true.
For an individual it is a dream come true to work for a large powerful organization, be it Microsoft or the Roman Empire, that gives you fame for free and covers up all your mistakes. It works for some time until eventually the sum of the covered up mistakes becomes too large for even the biggest organization to bear. It appears to me this is a pretty accurate description of “Imperial Overstretch”.

The amazing thing is that reality tends to be even more grotesque than fiction. Which Hollywood writer would even come up with the Pink Lade character? The whole Sidekick outage would probably have been rejected for any scenario as not credible.
I think the whole thing will be an inspiration for storytelling in the next decades which makes it even more fascinating to follow the events while they happen.

5 simonpie { 10.16.09 at 5:02 am }

That long horn thing looks really cool.

6 ChuckO { 10.16.09 at 6:26 am }

I think this story could have been told better with a little less hysterical glee and the Mary Jo Foley part is very tough to get through. This sort of reaction from the Apple community is ultimately self-defeating. I think your right that Microsoft is in a death spiral but time will tell. Please Apple community stop behaving like you want payback for the bad years learn to be happy that things are going so well. The best part of Apple’s success is that it’s proof that discipline and execution pay off in the end. Good products that are popular are the pay off not chastising Microsoft in shrill and desperate sounding tirades!

7 Sidekick data and ‘balloon boy’ - TotalCIO { 10.16.09 at 7:22 am }

[...] about whether a backup was necessary in order to go forward with the SAN upgrade. According to sources of blogger Daniel Eran Dilger, instead of doing a backup that would have taken six days, Microsoft management is said to have [...]

8 jaredgrubb { 10.16.09 at 7:25 am }

The MS press release seems to support your source that there was only one backup: “… a system failure that created data loss in the core database and the back-up.” (Not “the backups” or “backups”, but just “_the_ backup”).

9 commun5 { 10.16.09 at 7:36 am }

@ChuckO. I think there would be a little more understanding for Roz Ho were there not the suspicion that Roz, as the former director of Microsoft’s Mac unit, did nothing to scuttle the proposal/decision to remove Visual Basic macro support from Office 2008. This was a fiasco that was done solely for commercial reasons– to make sure that the new version wasn’t delayed, as if there were any significant innovations in Office 2008!

10 ChuckO { 10.16.09 at 7:52 am }

@commun5
I’m not offering any sympathy for MS or Roz Ho. I’m just saying the way Dan reported this was overwrought and that the Mary Jo Foley part was a bit of a mess. It seems like this was banged out in a bit of a shark-like feeding frenzy of glee over Microsofts gigantic blunder. Reporting this in a flatter style would have been much more powerful and would look less like the typical Apple fanboy sadness to reasonable people on either or neither side.

I’ve been reading Dan for about 2 years I think and I’ve really liked his well reasoned, deeply knowledgeable critiques combined with usually just the right amount of well deserved snarkiness. The posts lately about this sidekick issue seem to be increasingly and atypically over the top.

11 Blad_Rnr { 10.16.09 at 8:07 am }

I agree with ChuckO to some extent. MSFT is getting what they deserve. Win7 will not be a big hit and revenue will continue to decline. Miraculously the Mac is still gaining market share (it really defies logic) and the iPhone is selling like candy. Hammering MSFT is so easy it doesn’t matter anymore and arguing with Windows enthusiasts is pointless.

12 ChuckO { 10.16.09 at 8:23 am }

Apple doesn’t need what Microsoft has. If market share is the only measure of success then MS has nowhere to go but down and now that Apple is executing like a mofo they have nowhere to go but up. There shouldn’t be a single company making all the money. You can’t succeed being as big as MS. You can’t be everything to everybody. Apple is exactly where they should be RIGHT NOW they are kicking ass and taking names. MS peaked too early! Apple is executing at a time when amazing things are possible and they’re taking advantage of it!!! ENJOY IT!! It’s only going to get better. Let go of the past leave Schadenfreude to the Germans.

13 ChuckO { 10.16.09 at 8:38 am }

Schadenfreude(German) – pleasure derived from the misfortunes of others.

14 lmasanti { 10.16.09 at 8:44 am }

From the news…
“T-Mobile later said that it may yet be able to recover Sidekick users’ information. Those who do suffer permanent data loss will get a $100 “customer appreciation card” good toward T-Mobile service or products.
Not surprisingly, a number of lawsuits have been filed, including two in federal court in Northern California that allege both negligence and false claims on the part of Microsoft and T-Mobile. ”

http://news.cnet.com/8301-1001_3-10376498-92.html?part=rss&subj=news&tag=2547-1_3-0-5

15 stefn { 10.16.09 at 9:04 am }

I agree with ChuckO’s observations too. But here’s the deal:

The Microsoft media, pros and vols, is hard at work, even as Microsoft itself ebbs. It really isn’t Daniel who can’t let it go, it’s these folks. Honest to god, reading them is like listening to people who still believe that the British Empire exists or that the Dodgers will return to Brooklyn.

And their FUD, absorbed and spouted by the general media, is still a huge obstruction to millions of users simply trying to make reasonable buying decisions.

So if Dilger’s diligent efforts an an investigative writer requires a bit of comic relief and release, or even a touch of hyperbole to give an honest message a bit more volume, that’s just fine with me.

16 hmciv { 10.16.09 at 10:45 am }

“Bill Gates … proclaimed himself the Moses of tablet computing…”

What ever happened to the Moses tablets? Didn’t he release a set which was pretty revolutionary, followed by a quick upgrade…? I just don’t remember whatever happened to them.

17 seth { 10.16.09 at 11:17 am }

Wait, why would Mary Jo be upset with you Dan?

18 seth { 10.16.09 at 11:17 am }
19 gus2000 { 10.16.09 at 11:27 am }

Whaaa? Rabid Dan is “humbly admitting” being wrong, and is…apologizing?

Wow.

I thought being a blogger meant never having to say you’re sorry! Well good for you, Dan. Glad to see you man-up and take the hit for being wrong. That’s above what most other bloggers would do. Even the NYT prints their corrections where no one will see them.

See, that’s why I don’t blog, but instead comment on other peoples’ blogs where I can’t be held accountable. On the odd chance I do get called out, I can just change my nick and vanish to the ether of teh interwebs.

20 The Mad Hatter { 10.16.09 at 12:22 pm }

For those who wonder about Dan’s numbers for the settlements;

While the Comes vs Microsoft court case was going on, the lawyers for Comes placed on their website copies of all of the documents that they obtained during discovery that were not sealed. A bunch of people, myself included took it upon ourselves to download and preserve all of those documents. Someone even went to the trouble of setting up a torrent on The Pirate Bay to make sure that the documents were as widely disseminated as possible. Copies can be found in various places on the web including at groklaw.net

21 jdb { 10.16.09 at 12:43 pm }

In a similar vein as the Longhorn vaporware video, here is one on the vaporish Contour tablet.

http://www.youtube.com/watch?v=UmIgNfp-MdI

What’s interesting to me is the extensive use of handwriting. This makes it look like a leftover from the pre-iPhone era. But apparently Microsoft just can’t let go of the 1990′s tablet paradigms.

22 jdb { 10.16.09 at 12:51 pm }

Should be Courier tablet, doh!

23 The Mad Hatter { 10.16.09 at 12:54 pm }

But apparently Microsoft just can’t let go of the 1990’s tablet paradigms.

Of course not, Bill Boy says that Tablet Computing is the future, and so Microsoft is still committed to making it happen. They don’t seem to understand that Tablet Computing has already happened, and that its called the IPhone.

24 The Mad Hatter { 10.16.09 at 12:58 pm }

Ok, here’s the direct link to the Comes vs Microsoft documents on Groklaw. There’s also copies at Boycott Novell as text (the originals that I downloaded were in PDF format – I wonder if that’s why Microsoft decided to invent their own document format).

25 The Mad Hatter { 10.16.09 at 1:30 pm }

A final note on Comes vs Microsoft. The downloading the Comes documents by various people in the community proved important when, the second that a settlement was signed, they all disappeared from the lawyer’s website. It is suspected that this was part of the settlement, however we don’t know – it was sealed.

Since whenever Microsoft settles a case, they request that the documents be sealed (suspicion is that they pay extra for the privilege) a history of Microsoft’s misconduct is difficult to put together, and is mainly anecdotal, with the exception of the Comes documents.

26 pdeguise { 10.16.09 at 2:56 pm }

Working in the SAN/Enterprise storage/Data protection space, I cannot fathom any SAN vendor officially/explicitly stating that it’s OK to avoid doing a backup before an upgrade. It’s just not done.

It’s not entirely accurate to question why Microsoft/Danger staff couldn’t have done the SAN upgrade themselves. There are instances for every single major SAN vendor (IBM, EMC, HDS, HP, etc.) where particular upgrades cannot be done by non-certified vendor approved staff. This isn’t a level of accreditation that’s achieved by attending a course, but actually the certification that only the vendor or an accredited service partner will ever actually get. (E.g., in Australia as an example, particular EMC array updates are either done by EMC staff or, if sold by Dell originally, done by EMC, Unisys or approved Dell subcontractor staff.)

In short, there are some upgrades that customers aren’t permitted to do. The guts of the problem is inept backup standards in place and inept change control. It does however point out the untrustworthy nature of public cloud infrastructure at this point in time. I’ve made a recent posting on my blog (http://bit.ly/4xInTt) pointing out one way in which improvements might be made by introducing some compliance auditing and reporting for companies offering “cloud” services. Had this style of arrangement been in place, the Failure-That-Was-Sidekick would have been red flagged long before it happened.

27 ChuckO { 10.16.09 at 3:47 pm }

The Courier Tablet!!! Man o man, I can’t believe 9 to 5 Mac said they thought that looked cool. What a flippin’ mess that is! Let me open my portfolio. Tricked ya, see it’s a double-sided tablet. Oops dropped it trying to manage the beast. Again thank Allah for Microsoft.

28 Vaporware is Microsoft’s core competency « Sharing the truth one thread at a time { 10.17.09 at 7:07 am }

[...] market share has limits Vaporware is Microsoft’s core competency October 17, 2009 Roughly Drafted: However, Microsoft is also well known for advertising bullshit it can’t deliver. Bill Gates [...]

29 SamuRai { 10.17.09 at 8:37 am }

@ChuckO,

I think it is wrong to call Dan out for writing about topics which long-time readers of RDM are already aware; even if you object only to his apparently emotional style. Dan made it clear that he sympathizes with Microsoft’s Sidekick victims, and if you’re offended by the content then you are free not to read (aren’t you?).

I believe that people who don’t despise Microsoft for their history of unethical and criminal activity fall into four categories:
– those who are aware, but profit from those activities;
– those who are aware, but are agnostic or compelled due to the ubiquity of Microsoft’s technical plunder;
– those who have forgotten;
– those who don’t know.

I applaud the ongoing dedication of the few sites which contradict Microsoft’s propaganda through careful argument based on available facts (such as groklaw and roughlydrafted; apologies to those who maintain similar sites, but I have only limited time each day to seethe).

I also commend Dan for apologizing in a public forum, he must have a serious set of bollocks.

Respectfully,
dave (thoroughly corrupted Apple ‘fanboi’;)

30 The Mad Hatter { 10.17.09 at 11:11 am }

I believe that people who don’t despise Microsoft for their history of unethical and criminal activity fall into four categories:
– those who are aware, but profit from those activities;
– those who are aware, but are agnostic or compelled due to the ubiquity of Microsoft’s technical plunder;
– those who have forgotten;
– those who don’t know.

Those who are aware, but profit are criminals too.

31 The Mad Hatter { 10.17.09 at 11:12 am }

And talking about those who are clueless (don’t know) check this out.

32 Berend Schotanus { 10.18.09 at 4:30 am }

“I loathe that word, especially the ‘dogfooding’ verb form…”

The term “dogfooding” was completely new for my, I had to perform some Google searching in order to understand what was going on (it is amazing how effective Google can be). I think I can understand the bad feelings about the word: it really is a terrific specimen of ambiguous corporate slang.

The first part of it is something sympathetic, people should use products they make by themselves in order to know what their customers experience. This is obviously a good principle. It is likely to happen very often within a software firm that some staffers sends a friendly email to a colleague to attend him on a bug in the software he is responsible for. It might even be the case that in such emails less friendly words like “eat your own dog food” have been used.

The second part is that, when such wordings are becoming part of the culture, people start using it to their own benefits. In this case to cover up something I would rather call “protectionism”.
When the French government forces the postal service to use Renault delivery vans because both the post and Renault are state owned companies that would be considered protectionism. (it used to be common practice 30 years ago but is completely out of bounds today). The post company and the car manufacturer have nothing to do with each other, those people don’t know each other. When Renault delivers a poor performance and doesn’t react the only thing the Post manager can do is write a letter to the ministry which will probably remain unanswered.
Apparently the Microsoft management faces the same dilemma as the French government once did in how far they should go in protecting their own industrial divisions.
I think forcing a consumer products division to use specified business servers is not so different from forcing a post company to use specified cars.

33 The Mad Hatter { 10.18.09 at 9:48 am }
34 ChuckO { 10.18.09 at 4:54 pm }

SamuRai { 10.17.09 at 8:37 am }

I’m not sure you read what I wrote because I wasn’t doing any of the things you mention. I simply said that I thought the content of this article is too important for the overwrought quality of the writing and that the Foley apology was difficult to follow.

I’m (again) a fairly longtime reader of Roughly Drafted and an enormous fan of Dan’s work. I just thought these Danger/Sidekick articles were getting a little over the top in terms of reveling in Microsofts misery. This kind of behavior is unseemly in any like circumstance but especially coming from the Apple community and especially one of the Apple communities smartest and most articulate voices (Dan) it is counter productive.

35 The Mad Hatter { 10.18.09 at 7:28 pm }

ChuckO,
Hey, Dan wasn’t that bad, this one makes Dan look tame – Microsoft renders Sidekick data completely secure.

36 Lawsuits Against Microsoft Turn to Class Action Lawsuit While Microsoft Mobiles Become Dying Breed | Boycott Novell { 10.19.09 at 5:59 am }

[...] Linux is gaining, sometimes at Windows’ expense. Our reader Will concludes by pointing to this post about Pink/Danger, adding that “it doesn’t inspire confidence in Microsoft as a cloud vendor. Especially [...]

37 Data loss for Sidekick users Part 3 « Robby Pedrica’s Tech Blog { 10.19.09 at 9:52 am }

[...] And so the saga drags on almost 2 weeks after Sidekick users were left without their information. Daniel Eran Dilger at RoughlyDrafted Magazine has written a biting piece on the saga – although he may be a bit hysterical about it all, the fact is that this problem happened [...]

38 mysocialbrain: 19-10-2009 : protagonist { 10.19.09 at 1:10 pm }

[...] sidekick and roz ho it’s all her fault [...]

39 JasonBelec { 10.19.09 at 5:10 pm }

I laughed so hard reading this I ….. myslf!

You just gotta love MS, nothing, absolutely nothing holds them down, ever!!!!

They are totally comfortable in the deficiencies that define them.

40 c0t0d0s0.org { 10.20.09 at 5:43 am }

About that data loss at Microsoft/Danger…

Yesterday the usual news outlets speculated about the rumours circulating about the reasons for the data loss at Danger. For the uninitiated: Danger is the the company behind the Sidekick.

TGDdaily started by reporting the information provided by a …

41 More on the Microsoft / T-Mobile data outage | abram's nickels { 10.20.09 at 7:39 am }

[...] Link: Microsoft’s Pink/Danger backup problem blamed on Roz Ho [...]

42 nikwax { 10.23.09 at 1:14 pm }

what is even more ironic in that M$ is blaming Oracle and Sun for the outage, they are trashing the very people who are actually recovering the data:

http://www.appleinsider.com/articles/09/10/21/sun_oracle_save_microsofts_pink_after_danger_data_disaster.html

43 The Mad Hatter { 10.27.09 at 8:35 pm }

Nikwax,

Of course. The Microsoft backstab is a well known maneuver. Just look at their history, they do it allllll the time.

44 History of Microsoft’s “Slog” Against Android Brainchild | Boycott Novell { 01.10.10 at 5:10 am }
45 Why Apple can’t be too worried about Android 3.0 Honeycomb tablets taking away iPad sales: Part 2 — RoughlyDrafted Magazine { 02.09.11 at 6:55 pm }

[...] Major developer turns attention to Google’s Android How Oracle might kill Google’s Android and software patents all at once Microsoft’s Pink/Danger backup problem blamed on Roz Ho [...]

You must log in to post a comment.