Idosyncratic Code? #amazonfail
This just in from Mike Daisey, who commented on my “This Is Not A Glitch” post: a source inside Amazon tells him it wasn’t precisely a “glitch” but a translation error.
From an email exchange Mike forwarded to me between himself and Anthony Hecht, a reporter at The Stranger (I have Mike’s permission to quote):
From: Mike Daisey *email redacted*
Date: April 13, 2009 2:00:34 PM EDT
To: Anthony Hecht anthony @thestranger.com
Subject: Re: In case it gets lost in the comments…Well, this is the real story: a guy from Amazon France got confused on how he was editing the site, and mixed up “adult”, which is the term they use for porn, with stuff like “erotic” and “sexuality”. That browse node editor is universal, so by doing that there he affected ALL of Amazon. The CS rep thought the porn question as a standard porn question about how searches work.
The livejournaler is full of shit.
md
On Apr 13, 2009, at 1:41 PM, Anthony Hecht wrote:
> Ah, very interesting.. I would actually love to know a little more detail if you have it.. would be great to get the real story out there on Slog, especially as this livejournaler’s claim that he “bantown”‘d it for kicks is making the rounds:
> http://community.livejournal.com/brutal_honesty/3168992.html
>
> a.(from email)
Hmmmm…It sounds plausible. It still doesn’t explain how Amazon had a policy in place for deranking “adult” content all the way back in February, or why Amazon has refused to explain. More as this develops. And thank you, Mike!
ETA: Mike goes on to explain:
> Interesting.. kind of insane that all the international versions share that data, seems like exactly this kind of problem would have come up before..
It’s not the data they share, it’s the editing system–and not all the editing system, but weird, idiosyncractic bits. The Amazon system is mostly hand-built, and often super idiosyncratic, very Millennium Falcon meets Battlestar Galactica. It works, but it can be temperamental, and if you fuck up it takes awhile to correct.
(from email)
ETA: Fixed the thing that made this all wonky on LJ. Sorry about that. AND, Mike Daisey is a former Amazon employee; he just confirmed that to me.
ETA: In the comments, Mike says there may be no relation between deranking going on since February and the current whole-hog fiasco. I think the two are related because of the subject matter chosen each time to derank; and also because it fits the pattern I noted before with the robber-baron scandal.
Additionally, this has been bothering me: why even have sales rankings if they will mean nothing because of deranking bestselling (in their category) books?
Remember the Spore reviews fiasco and the new and old “review rating” systems too? Even if this is an honest mistake–and the evidence, the timing, Amazon’s response, and the targeting of certain subjects does not say “honest mistake” to me–it still fits into the pattern of Amazon taking the system that made them popular and trying to subvert it.
1. Amazon Censors Search Rankings To “Protect” Us
2. This Is Not A Glitch
3. Still Not A Glitch, But A Policy
4. (Update) Idiosyncratic Code?
5. Why I’m Bothering With AmazonFail
6. (Update) Seattle PI releases Amazon statement
7. Glitch, Ranking, & Porn
8. Days Later, Still AmazonFail
9. Glitch, Monoculture, Profit (AmazonFail Recap)
Related posts:
- Why I’m Bothering With #amazonfail
- Seattle PI has new #amazonfail statement
- Days Later, Still #amazonfail
Tags: amazonfail
April 13th, 2009 at 12:42 pm
Hmm. Yeah, totally. Very interesting.
And again LJ is being not so nice and making your lj all code-y. XD
April 13th, 2009 at 12:42 pm
“It still doesn’t explain how Amazon had a policy in place for deranking “adult” content all the way back in February, or why Amazon has refused to explain.”
It doesn’t explain that because these two events are unconnected–they may have been doing crap to certain people’s books in February, I wouldn’t know. I do know this is why thousands and thousands of books got delisted this weekend.
April 13th, 2009 at 12:57 pm
The real problem is that they are de-ranking ANY books, porn or not. If they want “family friendly” searches, they should have an opt-in filter for that. If they are going to carry a given book, they need to make it as available to search as any other book.
They have lost my business (and I buy a LOT from them) until they get rid of the de-ranking policy AND make everything available to all searches.
- Jesse
April 13th, 2009 at 12:57 pm
Still not buying it. Why would the Playboy Centerfold book, e.g., not be flagged, then?
April 13th, 2009 at 12:57 pm
So is Amazon UK differently coded then? Because it didn’t seem to be suffering from the search/ranking problems yesterday.
I find it very hard to believe that an online company like Amazon would have that large a codingfail embedded in their network. If that truly was the case, this should have happened far more often.
April 13th, 2009 at 1:10 pm
“Still not buying it. Why would the Playboy Centerfold book, e.g., not be flagged, then?”
I have no fucking idea. I doubt Amazon does either. All I’m telling you is what I told you.
“I find it very hard to believe that an online company like Amazon would have that large a codingfail embedded in their network.”
You are refreshingly naive.
April 13th, 2009 at 1:11 pm
It also doesn’t explain why children’s books such as “Heather has Two Mommies” were deranked.
April 13th, 2009 at 1:13 pm
A fourth explanatory trial balloon, and again it fails.
Unless one can explain how “Heather has Two Mommies” fits into any of these categories, even given a translation barrier.
Or a book on preventing gay teen suicides. Apparently books on preventing gay teens from being gay rank #1 when you search on homosexuality, but a books on preventing gay teens from taking their own lives are forbidden?
See, I’m more than willing to believe there’s a reasonable explanation here.
But I don’t think we’ve heard it yet.
April 13th, 2009 at 1:23 pm
That still doesn’t explain why the pro-gay and lesbian books got banned but the anti ones didn’t. Or why Stephen Fry’s biography got deranked but Ron Jeremy’s didn’t. No, this explanation doesn’t fly at all.
April 13th, 2009 at 1:28 pm
[...] in the time it took to transcribe all that, another internal Amazon source has come forward claiming that an Amazon France employee is the overz…. At this rate of blame-laying, maybe Amazon will propose by Tuesday that it’s all the fault [...]
April 13th, 2009 at 1:31 pm
This kind of simple error seems like something Amazon might readily admit to — but they haven’t. So I’m not yet convinced.
April 13th, 2009 at 1:37 pm
Actually, Jane over at DearAuthor checked out a bunch of the books that were filtered and discovered that it was due to them having keywords like erotica or gay/lesbian in the metadata ( http://dearauthor.com/wordpress/2009/04/12/amazon-possibly-using-category-metadata-to-filter-rankings/ ). “Heather’s Two Mommies” got filed under Gay/Lesbian literature, for example, whereas “A Parent’s Guide to Preventing Homosexuality” didn’t, so I can buy this explanation from that perspective.
What confuses me more, on the other hand, was that I was checking amazon.co.uk and amazon.de yesterday and I swear not all of the books deranked on amazon.com were deranked there. If someone on amazon.fr screwed up and the code affected all the sites, you’d expect the same books to be missing on each, right?
April 13th, 2009 at 1:50 pm
I’d buy this. Why doesn’t Amazon just…say something? So bad…so simple to just say “There’s an error. It’s wrong. We’re sorry. It will be fixed.” Just saying that would have satisfied at least half the people tweeting #amazonfail.
A big organization with big, gnarly coded added on top of code…oh yeah, easy for there to be problems with it. Not really glitches…but just things coded differently than if you could start the whole thing over.
It is still the dumbest filter idea ever to build a filter that relies on removing sales-ranks. It’s unfair…if something sells well, then it sells well, regardless of what it is. I get the need for a filter, but not the need for it to be imposed on all users. It should be optional.
I’m sure they had totally practical reasons for trying to implement the filter this way…less server load, less coding…fewer/smaller queries and conditions. But sometimes the harder way is actually BETTER.
April 13th, 2009 at 1:56 pm
Actually this makes a lot of sense from what I know about database code. Trying to apply the right tags and searching and everything to as many products as Amazon has across as many international sites as they have is a pain in the butt. I think its’ mostly been DUMB LUCK that it hasn’t exploded in their faces before.
I work as a quality assurance engineer in a company with a large database structure and do all the testing for the external websites and it’s amazing what havoc you can cause by editing one dumb field. From what I’ve heard the Amazon code base has been just one step ahead of the curve of their business, VERY TYPICAL, and so means there’s some duct tape, bailing twine and bubble gum wedged in there.
I think the bigger problem is in A: Not getting an official reply out sooner. Though it was a holiday weekend so very possible it was hard to move up the escalation list, which is what something like this would do internally. B: All editing of such things should be done on an internal test server FIRST not something attached to the live system. C: Service folks should NOT be giving out answers to why something happened unless there is an official company statement, in which case they should refer all questions to the statement. Getting quotes about this is a policy, this is Blah blah blah, only further muddies the waters.
~J
April 13th, 2009 at 2:22 pm
This still doesn’t compute, and the explanations being leaked are a combination of scary and utterly unbelievable. As Romeo says, if it was a “glitch” in .fr, why was mainly .us affected. Incidentally, the area mostly affected is also the place where there are a lot of groups trying to get exactly these types of books removed from Amazon. Sorry, that seems too much of a happenstance to take seriously.
Also, and this is a big one for me, if this was a glitch, Amazon should instantly, I think, admit to malicious intent on their part. If not they are really saying that “oh, some kid in France who doesn’t know the difference between ‘adult’ and ‘child molestation’ could easily compromise all of our software, and by the way, please do trust all of your corporate data to our cloud-computing services”.
I seriously hope it is a little harder to mess up the software that runs the business of Amazon.
April 13th, 2009 at 3:04 pm
Having looked at the metadata on a bunch of books that were stripped of their rank, as well as the metadata on books that weren’t, this doesn’t fly. Plenty of books with “sexuality” in the metadata (including the infamous Parent’s Guide to Preventing Homosexuality) were left up, while books lacking that terminology (but were labeled under Gay & Lesbian) lost their ranks.
April 13th, 2009 at 3:10 pm
OMG, that was the last cliche I had not yet seen on this topic: it’s the French’s fault!
Yeah, right. Go sell your Freedom fries elsewhere, I am not buying it. Seriously, what kind of idiots do you take us for?
April 13th, 2009 at 3:38 pm
Actually, it would totally make sense that it didn’t happen to all sites at once. A large network like Amazon would not update in real time, but rather in batches and there would be server caching.
April 13th, 2009 at 4:13 pm
It takes time to track this kind of weird crap down. Does Amazon owe everyone an investigation and apology? Yes. Might it take a day or two to figure what the heck happened? Yes.
April 13th, 2009 at 4:14 pm
This doesn’t explain the suggested searches you get when you go to Amazon.
If you type in “homosexuality” as a search, it suggests as search terms:
homosexuality and the bible
homosexuality and the politics of truth
homosexuality and christianity
homosexuality and the catholic church
homosexuality and american psychiatry
The first titles that show up in a search for homosexuality are:
A Parent’s Guide to Preventing Homosexuality
You Don’t Have to Be Gay
For the Bible Tells Me So
Can Homosexuality Be Healed?
All of the suggested search terms above bring up right-wing screeds against homosexuality; the one on “homosexuality and american psychiatry” brings up books from the 1980s about how the APA was “politicized” into delisting homosexuality as a disorder.
If you type in “gay” as a search term, it suggests “gayporn”, “gaydvd” and similar variants which take you to their dvd/movies section and doesn’t show you any books (like “Gay New York” for example).
So are these suggested searches completely generated by how users search at the site? If so, it seems odd that “gay” doesn’t bring up any “normal” phrase starting with that word and that “homosexuality” suggests right-wing BS for you.
April 13th, 2009 at 6:31 pm
I note that you don’t include the suggested search term “homosexuality 101.” Does it undercut your paranoid view of the matter?
And yes, the search suggestions are derived from the way in which users search the site. You don’t really think that Amazon, one of the most gay friendly and understanding companies in the world as well as being based in the very gay friendly city of Seattle has some gaybashing executive sitting around thinking up search terms, do you?
Use your head, for just one moment, and you’ll see how paranoid and overly victimized all that makes you sound.
Completing the suggested term gay with “books” will take you to plenty of gay books. The first title returned is “Between Men: Best New Gay Fiction”
The system does not know when you type in gay at the all departments whether you want gay porn, gay books or gayle collins.
April 14th, 2009 at 4:06 am
I’m not as concerned about Amazon as I am by groups of users that may try to “game” the system through a concerted effort involving automated scripting and the like.
Paranoid? Let’s just say I’m skeptical of systems that are so user driven that they can be taken advantage of by someone with an agenda. There’s a small group of fanatics on the right that has a long history of pulling books from libraries or setting ridiculous standards for local school boards.
It’s not unlike attempts to manipulate Google page rankings or other misinformation campaigns.
April 14th, 2009 at 2:31 pm
[...] to Daisey’s inside sources, “A guy from Amazon France got confused on how he was editing the site, and mixed up [...]
April 15th, 2009 at 1:02 am
[...] void of any real information, various theories have been floating around, some quite creative: confused French programmer, Internet troll, right-wing conspiracy, etc. A storm on Twitter and Facebook on Sunday led the [...]
April 15th, 2009 at 10:09 am
“From what I’ve heard the Amazon code base has been just one step ahead of the curve of their business, VERY TYPICAL, and so means there’s some duct tape, bailing twine and bubble gum wedged in there.”
That’s pretty much it. Based on what I saw during the year that I wasted there, your statement is actually sugar-coating the reality of Amazon’s abysmal codebase. In reality, I find it hard to believe that it works at all, and having been there, I saw how many things failed routinely.
Amazon uses developers as basically babysitters for their systems, without which none of it would work.
“I think its’ mostly been DUMB LUCK that it hasn’t exploded in their faces before.”
It’s not luck, it’s lots of developer with 24×7 pager duty.
People like you don’t exist there; QA is a foreign concept to someone who just graduated from college and landed a job with a big internet company, and that is exactly the kind of person that Amazon has writing its most critical code. Amazon takes full advantage of the fact that those kids are willing to carry pagers and feel like heroes because they fixed some major issue at 2am and yet show up for a full day of work the following day, because “that’s how it’s done” there.
When I was interviewing, almost every single company that I interviewed with (around 20 phone and close to a dozen in-person) had at least one ex-Amazon employee in the interview loop, and not a one was surprised that I was looking to get less than a year after joining. (In fact, one of my co-workers at Amazon said that I’d leave soon, because the best ones always do (my co-worker’s words almost exactly) — around three months after I joined.)
April 17th, 2009 at 7:44 pm
[...] of Amazon… specifically, it may be an editing error by a single employee. There are still lingering doubts and inconsistencies for some. Dear Author brings up past episodes of similar manipulations. Vroman’s [...]
April 20th, 2009 at 2:58 pm
[...] For an inside look at Amazon’s in-house reponse, look here. And check out Lilith Saintcrow’s blog for the nitty-gritty here. [...]
April 26th, 2009 at 11:24 pm
[...] explanation at the moment seems to be a combination of bad coding and bad PR. Lilith Saintcrow posted this from an insider at Amazon: “Well, this is the real story: a guy from Amazon France got [...]